"I've always been used to going back to work on the 3rd or 4th of January," the PM said before Christmas.