I don't use paper towels or dryers, but I do wash. I just don't want to touch something that thousands of unknowns have touched right after I wash my hands. I use my sleeve to open the door and then air dry.
I recently read on some baby-site (cuz I has one so that was the scope) that new studies have shown that bacteria is less often transferred from parent to child when they wash and dry their hands often rather than if they just wash and don't dry. Don't remember the reasoning for it, though. It just came to mind.
My only guess is the moisture still on your hands brings a jerkstore full of germs back as soon as you touch the door knob or whatever else is in the bathroom. Dryer hands might not attract as much upon contact with various objects. I think its a good idea to cycle towels into your laundry at least once or twice a week though.
Also, I always use anti-bacteria soap which I definitely recommend 1) because it kills most of the germs and 2) because I work for a soap company.