A 41-year-old fisherman has died after being swept off rocks north of Sydney - despite wearing a floatation device.