The former judge has said he thinks it will be ‘better for them to find two women to replace Tess and Claudia’