The Scottish government said Swinney "politely declined" the offer to join the event as part of the King and Queen's state visit.