Lord Spencer Livermore became the first serving minister to publicly endorse overturning the 2016 referendum vote to leave the European Union.