Marketing 101: Face-to-face personal contact beats anonymous literature of any type, any day of the week. You are doing the right thing.
Yes, some contacts will be turned off, just as some are turned off when you leave fliers (as is discussed elsewhere on this site). Those people were already in your "NO" column. Others will see what I see, which is someone with a good work ethic and courage, someone with who will communicate with them directly from the get-go. I suspect customers you gain this way will remain loyal customers longer than customers you gain through anonymous fliers.
That's the encouragement part. Now for the advice part:
Turn some of those NO's into future YES's. Always thank them for listening to you. Always ask them to accept your card in case they need you in the future. They may really mean NO, then find out they need lawn care a few weeks later (broken leg, new job requires travel, any number of reasons). They will remember you and be the first that they call.
Always ask them to help you by keeping 5 of your business card to give to friends and neighbors if they should encounter anyone who needs lawn care, or in case they should need it in the future. People (generally) won't mind if you ask them to help this way and you'll be surprised how many will rise to the occasion.
If you get referrals this way, always call the referring person to thank them, and add them to your Christmas card list, even if the person is not a customer.
Good Luck. I believe you will do well.