you should never be afraid of growth. If you are, you dont have systems in place. A lot of people say its easy as a solo operator, I think its harder, because you have to do all the work. With systems in place, you can have others working while you handle the business side or the aspects of your business you enjoy the most. I know what I am describing is the ideal situation, but it is obtainable. In most companies, the guy at the top usually has the easiest and most fun job. Why? Because he worked his tail off to build it, then got people in place to run it.
You need to put systems in place from the beginning, so that as you grow, they are already there. Its a lot easier to put in systems with 1 employee when your doing 40K a year then it is at 5 employees doing 400K a year. Good luck and dont give up