I believe that I would have told the contractor from the start that his customers wanted to switch. If he wanted to sell them, that's fine. If not, maybe I would have agreed to continue doing them and paying him a referral fee, comparable to what he was getting when you were the subcontractor.
I agree that it is not worth burning the bridge. I was at a software/hardware company where one of the largest businesses in the world took a liking to us and helped us out. In time, we decided that we were too big for the partnership and really screwed the other company. It was ugly and in the end they nearly ran us out of business. They underbid us on countless contracts where they would have partnered with us in the past and they managed to create some real supply problems for us via some of their other partners.
I'm not saying it will be this bad for you, but there will be enough challenges out there without a competitor who has a personal grudge.
Why not just ask him how he would like you to handle it?