Son of the late Billy Graham says he ‘didn’t jump to the same conclusion’ as others after bipartisan backlash