Opposition parties walked out of a council meeting about immigration after labelling it a “divisive fantasy debate”.