Residents in a Norfolk hamlet have hit out after councillors approved plans to tear down a cottage for what has been likened to a turret.