James Glen said he supported partner Linda Holt's decision to withdraw from the election race on Thursday