The Duke of Sussex is said to be sad that his children 'are missing out on the extensive family network'