I don't even try to get a scaled drawing going when I'm on site. Other than that, my process is similar to yours. I measure to locate all windows and doors on the house, stoops, decks, patios, and walks. Then I flip the page and get trees and other critical points not attached to the house. Then I shoot grades (unless it's a planting plan). If it's an easy lot I'll grab 8-12 spot grades, otherwise I lay out a 15'x15' grid in the area I'm designing. Then it's lots of photos with the DSLR.
I generally spend between 10-30 minutes measuring a site unless there's a lot of complexity. Then I go back to the office and basemap it in AutoCAD, starting with the plat if possible. That makes the question of rainy/windy days a lot easier - I just need to keep the sheet dry long enough to write my notes, then I'm out of there.