In a quaint English town, the Victorian architecture and historic streets have become the backdrop for TV stardom