Four members of the Adelaide Festival board, including its chairwoman, have quit following a mass boycott from speakers.