The Labour MP for the village said families were 'moving away' because of the disruption caused by HS2