A row has broken out over plans to build a ‘granny annexe’ in a North Yorkshire garden over concerns about the impact on pedestrian safety.