Lord Mann’s warning comes as controversy continues to swirl over the Maccabi Tel Aviv fan ban in Birmingham