It was an area that 'once had prostitution issues 20 years ago' but now residents say people are 'flocking here from more expensive areas'