I spoke to more than 20 attendees, and many said that without Buffett as the main event, the meeting could lose its sparkle.