I think it depends on how your business is set-up and your expectations and goals. Since your even considering dropping clients, I'm assuming your not based on cash flow to cover payments/overhead etc..... And more based on profit/time and efficiency.
That being said, it's a simple question. Is the outcome/income from these clients worth your time and energy? Simple sure, but perplexing at the same time. Do a simple spreadsheet keeping them, and do one without. I tell people to put a price on their perceived and/or real headaches they have, and then do a simple P&L statement.
If the $ is not worth it then drop. If the $ is then deal with it from a different approach, or else keep them till you replace them with another client.
This is a hard question to help with without knowing your internals. Good Luck