We've always had much better response with flyers than mailers. Mailers produced only marginal success this year and I doubt I'll do it again. Further, they were 3 times as expensive as our flyers are. So, per dollar, mailers were a LOT worse.
Your success will depend on the time of the year you lay the flyers. If I laid 10,000 flyers this time of year, I'd be lucky to get 10-20 calls. But If I laid 10K flyers in April or May, I'd have more calls than I could possibly handle (200-300 calls I bet.) During the summer or late summer I'd maybe get 40-50 calls per 10K. Of course, that's just for my area. YMMV.
Flyers is really the best ROI of any kind of marketing. You can't beat it. But the challenge is nobody likes passing out 10K flyers. Especially in hilly areas (and that's where all the nice homes are in our area).