Counter-protesters far outnumbered anti-immigration activists at a planned rally outside a former barracks.