A Malaysian court on Monday dismissed jailed ex-premier Najib Razak's bid to serve the remainder of his sentence at home, saying a royal document authorising the move was invalid because it was not made in accordance with procedure.