Authorities have clashed with protesters at the site which eventually be the centrepiece of the Brisbane 2032 Games.