Do you want to try and get rid of them permanently or are you just trying to cut them back?
Cutting them back: Any time is as good a time as any. You'll probably especially want to cut them down should they sustain freeze damage in winter time.
Removing them: Good luck! Nearly impossible... But, not quite impossible.