ITV's plan to poach Graham Norton from the BBC is said to be in chaos after his new show The Neighbourhood suffered plunging ratings.