Usually put a thin layer of mulch down over the old, it's only getting put down to incorporate the pre-emergent as well as add color. However, after a few years, you will have built up enough to where you should consider removing old.
If it's getting piled up on tree trunks, remove it. If it's within a couple of inches of house siding, remove it. If its ON house siding, you better remove it PRONTO and hope no critters have decided to use the mulch as a superhighway into the house!
Dan