The part that sticks out to me most when I read your post is that you "....didnt keep on top of it or I lost the piece of paper with important info on it ect. Basically there is no consistancy which is starting to hurt me..."
I think I'm close to being in the same boat as you are. But I focus on staying EXTREMELY organized. It takes time, no doubt. And I have help on the accounting end- but I still have to estimate, invoice, prep FOR the accounting and reconcile all receipts and invoices for materials, etc... And work in the field everyday. It takes dedicated organization everyday (clipboards, truck organizers, updated "to do" lists on the computer, getting records and notes onto the office desk everyday!)
My issue, as I think yours is, is deciding when to get OUT of the field and into the office to work "on" the biz and not "in" it. That happens with more labor, more equipment to keep those additional guys working, and so on.
Not too familiar with Gopher, but my Quickbooks really helps with customer lists, estimates, and invoicing. We don't have it set up to track any real money, but I can easily keep up with payments and "pending" monthly invoicing, as well as one-time jobs (of which we do a lot)