The retired founder of L Brands said though he never circulated in Epstein's social orbit, he was wrong to put his trust in the disgraced financier.