Simon Barr said he feared he and his family "were going to be deeply penalised for the successes that we are now getting from the businesses".