Residents have spoken out against plans which would see a derelict town centre church turned into a community hub featuring homeless accommodation.