I planted some this spring even before frost was over and I had some brown leaves and some that were frost damaged but they bloomed just fine, they bloomed like crazy but as summer wore on after the blooms matured they started to turn brown, wilt and die off.
I was questioning what to do just like you so I did some reading it's perfectly okay to pull off dead ones, I watered as needed throughout summer and low and behold new blooms started popping up after I pulled tons of dead ones off. The plant looked terrible due to the weak wood the blooms were weighing the plant down something terrible so after pulling off the dead ones it came back to life.
I will from now on pull off weak brown flowers so new ones can bloom. Here is what it looked like during early summer. I'm figuring the old flowers were sucking the life out of the plant it got much healthier after the old ones got removed.