I usually carry $200 - $500.
When you get mugged and lose $500 I'm going to be laughing.
I will say that it would be a good idea to carry at least $20 at all times just in case.. but I just never go to the bank to get cash out. I use my card for everything. If I get mugged, I can just cancel the card and have them send me a new one.. Just makes sense to me.