Australia's social cohesion is under strain, with online outrage, falling trust and rising political violence posing growing security risks, a report warns.