There’s something truly refreshing about a town that refuses to reinvent itself just to please the tourist crowd