Four ministerial aides have stepped down and more than 60 Labour lawmakers have publicly called for him to resign.