I think the hardest things to give up are the guaranteed paychecks with annual raise (unless I for some reason get let go), the VERY cheap full cadillac of plans family coverage health care package with a $1200 debit card to cover all the copays, and the retirement plan which is I give 3+% of my income and they put in 8% extra. Yes you read that right, I put in 3% of my salary and they "match" it with 8% of my salary.... I mean thats like free gambling money for me in the market with my average rate of return being 9.6% so far. Free money to make money. This is what keeps my planted at this career not to mention I'll have EVERY holiday off and very light summers, hence the part time landscaping. I guess maybe I've answered this for myself and will most likely stick to part time and maybe even part time with summer employees. Make extra side money and delegate all the back breaking work to summer help..
Anyone that has a well paying job, great health benefits that you retire with, you have to be crazy to walk away from that. You can always retire then expand your part time route.