The charming market town boasts 75 independent shops, famous gardens, and a castle used as a filming location in Harry Potter