Local councillors and residents have expressed concerns that an influx of English people will decimate Welsh language and heritage.