Americans who feel “betrayed” by the president are drifting away as the midterms approach, a polling expert tells Newsweek.