Thank you for contacting Toro with your belt wear concern. Belt wear/failure is usually related with belt tension and/or alignment.
When a belt does not have enough tension the belt failure is unusually a pulling or tear appearance to the belt when it fails.
When a belt has too much belt tension will have a melting appearance or hard coating from heating and softening then the material cooling back down.
When alignment is not right you can see wear on the edges (fraying) and/or the belt will not stay on the pulleys and fall off. What you have described it has the symptoms as if to much movement in the idler pulley assembly. I do not know what the exact spec is on the movement but the amount you are experiencing seems excessive.
We have attached the Toro Parts breakdown for your reference to assure all the parts are in place.
The Toro Company