Scottish Labour leader Anas Sarwar admitted it hurt him personally to call for Prime Minister Sir Keir Starmer to quit.