While the current presenters drew up a 'pact' to leave together, there is no deal agreed between the remaining original judges to go at the same time