They lost no fewer than three appeals and now planning inspector dismisses 'family circumstances' argument