Ministers have voiced frustration that Lord Hermer is exploiting his close personal friendship with Keir Starmer to bypass normal Cabinet government.