Works to transform a central park into an Olympic stadium are set to begin amid strong opposition from the Indigenous community and local residents.