Shadow finance minister James Paterson has called for the Prime Minister to apologise over the matter.