His comments came as SNP leader John Swinney said he expected his party to go on and win a record fifth consecutive term