I can appreciate all too well where your coming from, and I honestly don't feel there was much more you could have done in that scenario. I've been there plenty. It reminds me of a couple of times when clients insisted on having their Interior Decorator involved in the process. It just doesn't work, to put it politely.
I haven't charged an initial consultation fee in my 21+ years of doing my own contracting. If it works for some, I can only respect that. For me, my strategy has always been somewhat different. For the record, I've squeezed out competitors because of their up front fees. The only reason I know that, is because it comes out in the consultation process. This is the part of the process where I let my client vent, so to speak. It doesn't mean I agree w/ the client, it just means that for some reason the other contractor "rubbed them wrong" during the most delicate part of the "building trust & credibility" interview. Again, I have no problem w/ those who do it...I just don't feel that it works to my advantage.
I'm not clear on why clients have a problem w/ it, but if I had to guess, I'd say it's mostly the timing part of the contractor. Often times, contractors can be overconfident and do come across as being full of themselves...I've heard that a lot. In other words, they didn't spend enough time somewhere w/ the client, that the client doesn't yet feel ready to make even the smallest of commitments. In the back of their mind there are walls going up because all they heard was $$. In the mean time, the contractor can talk about the project for the next hour, while all along the client is still thinking about that fee/fee's/rate per hour...and not a word registered there after, as to what the contractor was saying.
My point w/ that would be this: If you feel the need to charge a fee, then by all means do it. Just realize, the timing part is a very delicate one. My clients will have a reasonable idea, as to what things cost before I begin the design process. They will have a range, as I don't lock myself in on anything. Their response will dictate where things go from there. If I feel them pulling away from me, then it's time to back up. If that process exhausts itself, I have to decide if it's time to move on.
My final point would be...I wear them down. I measure/gage what direction things are to go, based mostly on there enthusiasm and the comfort level that I feel has been established. I will take as much time as necessary to address their concerns and develop my repore w/ them....comfort level.
I honestly don't know of too many who will go to the length that I will, which makes it that much less difficult to sell. The client, in many cases will not follow through w/ the others, for whatever reasons. All I can say beyond that, is I will be the last guy in the door...even if it means creating a "loose end" to get me there. That can be a revision of the plan, a pic of some material, whatever it takes.
I don't really chase the small stuff so much unless it's practically handed to me, that's where there's usually too much traffic & sifting to do, and quite honestly will give me a headache. I work harder, much harder at the complex projects that would typically cause the client to utilize 2-3 contractors, instead of one.
By the time we are even mid way through my process, the client simply doesn't have the time nor the interest in going through the process again w/ someone else, let alone 2-3 more contractors for the other work that the other companies might not handle. If they did, they would realize that they would risk not getting things done until the following season. Chances are, one won't be happy about it and the other will finally give in and say, "fine lets just do it." By now they are probably sick of the mud that the dog & kids keep tracking into the house. The price is no longer so much of an issue, the only issue is, "when can you start."
Didn't mean to write a book, but that's a watered down version of how things may go in my world. Great discussion and ideas on here otherwise.