"He obviously didn’t get hugged a lot," the late-night host said of the president in a new interview with Vulture