Kimora Lee said it was tricky to buy back Baby Phat in 2019 after walking away with $20 million from selling it.