lol, nope you don't have to take the spindle off.
The "pole" that is next to the pulley is a belt "finger". As you said, it keeps the belt from coming off. The bolt should go through the deck to the underside. It is adjustable, and you will need a socket wrench(es) to loosen. You can then swivel it out of the way to get your belt on.
Normally, they are set so you can get your belts on without too much trouble, like just pulling it with one hand while cramming the belt in with the other, but maybe not the case with yours.