Angry residents have banded the potential removal of a beloved and rare landmark from a town centre as “disgusting” and “ridiculous”.