“I guess there are people who really don’t want us here,” the club's vice-commodore told the Toronto Sun