Barnaby Joyce has singled out one group for the shutdown of Australian oil refineries, including during his tenure as deputy prime minister.