The family of a former governor-general and archbishop has described how he became a "lightning rod" for institutional failures to prevent child sexual abuse.