Several homes were demolished on the land before it became a site office to serve a nearby 200-home estate