For the workers I'd go with a darker color, just to keep them cleaner. Khaki gets dirty real easy. For the managers / estimators I'd stay with Khakis and also make sure you're wearing a colored shirt.
In our company there is one dress code for workers and another for management/estimators. For one, it's more professional that way. For two, I have always believed that the client or anyone walking or driving by should be able to tell, just by looking, who the project manager is on any given jobsite. The more professional and clean look of the project manager sets him apart from the rest of the crew. It also helps create a little more respect from the workers too.