The area was once used for oyster fishing, and Victorians tried to cash in on the boom of seaside holidays, but the sea had other ideas