Huge waves hit hundreds of swimmers and locals say the bad weather meant the event should never have been allowed to go ahead.