The historic mansion was being considered for a £19m five-star boutique hotel development just four years ago but is now crumbling