Some have said their US travel permit has been revoked for no apparent reason just days before the World Cup