A former tennis ace turned Liberal MP has taken aim at members of his own party in his decision not to contest the next election after a "challenging period".