Firefighters have revealed buildings connected to the culturally significant site have been lost in a raging bushfire.