A bid to extend the opening hours of a high street restaurant has been refused over concerns about the impact of noise on neighbours.