He told opposition councillors to let him 'get on with the job' in the face of accusations of 'incompetence'