Traders in one of the weakest ranked high streets nationally are “really worried” at the rate of decline - and fear the worst may still be to come.