Authorities have clashed with protesters at the site, which eventually be the centrepiece of the Brisbane 2032 Games.