Violence, drugs and understaffing at a remand centre are a constant safety threat and politicians must commit to returning it to public hands, a union says.