Former Fed Chair and Treasury Secretary Janet Yellen said she was "surprised the market isn't more concerned."