Local residents have shared their anger over plans to build more than 3,000 new homes between conservation areas and listed buildings.