Home secretary ‘no longer has confidence’ in police chief who banned football fans