The prime minister may have painted himself into a corner by refusing to call a royal commission into Australia's worst massacre in decades, an expert says.