Critics claim "flag signalling" has risen in the area since a backlash against plans to house up to 70 asylum seekers nearby