Concerned residents living in the heart of a housebuilding hotspot say they are considering leaving Kent due to the amount of development in the area.