The Backstreet Boys made the "right decision" not signing with Shaquille O'Neal back in the day ... so says AJ McLean, who tells TMZ Sports while he adores the NBA legend -- he doubts they'd have the same success had they inked a deal with him.…