A bid by an Australian neo-Nazi group to become a political party has been ruled invalid as a High Court challenge looms.