When I started mowing, I used to do it part time as well. I started around 3 or 3:30 and would go until 5 or 6. Although it was only a few hours, I scheduled accordingly for 1 reason...when the late fall hits and it starts getting dark at 5.
I also wanted to set a cut off time for my customers. Whether its them eating outside, watching tv, putting kids to bed, etc, my customers shouldn't have to be bothered by lawn mowers and loud noises at night. I remember as a kid, I had a 7pm bed time. If a lawn crew showed up at my house at 8pm to mow, I bet you my parents would flip out!!
But as far as timing of mowing "high end" homes....it doesn't matter at all. Just be respectful of them and the time your showing up.