Just 24% had a "favorable" opinion of the three-term congressman after conversations leaked about his alleged affair with a late aide.