Michael Rapaport's explosive comments before his banishment pissed off "Traitors" cast members ... which was par for the course while shooting -- because we've learned they couldn't stand the guy the entire time. A source close to the production of…