Proposals to host a three-day hot air balloon festival every June - but residents fear for their homes