Traders in one of the hardest hit high streets in the country are "really worried" at the sharp decline in footfall