I would leave it on for spring cleanups.
I'm a bit puzzled by your statement that it isn't easily taken off/put on. I leave the rear hopper on my mower all of the time and put on/take off the blower and tube to the rear hopper on a regular basis. Usually the original installation takes some time but after that putting it on and taking it off is pretty simple and easy. What make/model is your mower.