Philip Ruddock, the father of Australia's controversial asylum seekers policy, says the Opposition shouldn't fixate on numbers.