How do airlines set bag and weight limits? An ex-pilot explains

As a former pilot turned aviation safety researcher, clearer rules for carry-on baggage are welcome, not least because too much cabin baggage is a real safety concern.