The failed voice referendum gave Australians permission to be racist, which continues to cause distress for Indigenous people, a health conference has heard.