Wall Street’s $13 trillion ETF machine pushed boundaries this year, cranking out ever-riskier products to feed retail traders hooked on yield, leverage and novelty.