Coastal and rural communities would benefit from having a dedicated government minister, councillors have said.