Some investors attracted to the asset class are finding an unexpected downside to long-term investing