The rejection diminishes prospects of an industry-reshaping deal that would face tough regulatory scrutiny.