Around 20 squatters were believed to have broken into Raffles in Chelsea, a venue once frequented by Prince Harry .