Less HP on some 32"
No Hydro option on most 32"
Narrower drive tires
Probably can't use a sulky on a 32"
On some same brands, a 32" deck takes the same blade as a 48"
Same amount of "trim" usually (deck overhang from rear tire)
Productivity loss is marginal based on most lot sizes that you would us a 36" on.
If using belt drive models only, a 32" is a little lighter therefore it can be manhandled easier. Less wear and tear on your body (although you might have an extra pass or two).
I am sure there is more.