From hosting secret meetings during World War 2 to being a former Royal residence, one overlooked estate has a remarkable history