The prime minister has found himself at loggerheads with a former Australian of the Year over an 'unfortunate' one-word remark.