Several large investors say it was hard to arrange meetings with Albert Manifold before his removal from oil major