An abandoned sports centre set to partially reopen this summer has suffered “significant damage” after vandals trashed the site.