A convicted phone hacker working for Prince Harry and other celebrities tried to blackmail a Mail on Sunday executive, a court has heard.