Police have been called to Victoria Park, the future site of the Brisbane 2032 Olympic stadium amid protests.