The grade II listed former restaurant has been left vacant since 2014 and is becoming an eyesore in a leafy village