Jewish groups have hit out at Anthony Albanese’s rejection of a federal probe, claiming the Prime Minister needs to “listen to the people”.