Experience with "Street Folk"

I was down in New Orleans Saturday and was walking down Loyola, specifically near City Hall, towards Poydras. My SO and I were holding hands and we're heading towards the Smoothie King Center.

Two guys who looked dirty, but not exactly homeless, approached us from the opposite direct. One fellow was an elderly black male and the other looked like a slightly younger Hispanic male.

They walk up and start trying to chat with us and asked if we are going to the concert. I admittedly thought the Hispanic male asked if we we're going to party (the two of us were dressed fairly nice) and thus further justifies the remainder of my story.

Now let me just clarify that I dislike conversing with most strangers and will avoid any sort of interaction at most costs. I don't like to casually bullshite with people I meet on the street and I usually have a pretty good sense about knowing who to avoid.

Plus, I'm well-aware of some of the stuff that people down in New Orleans, or in the center of any major city for that matter, will try to pull. Honestly, my initial impression was that the two aforementioned males were panhandlers of some sort so I tried to avoid them.

When one of them asks if if we are going to the concert my SO foolishly replied 'yes'. Personally, I wanted her to take my cue and just ignore them. Meanwhile, I simply responded "Nah, man," and kept on walking down the sidewalk.

Apparently, the black gentleman took great offense to my unwillingness to chitchat and responded with a slew of expletives including:

"frick you, you pussy arse bitch."
"You frickin' piece of shite n***er" (I'm white by the way)
and "You better hold her hand...(something incoherent")

Was I in the wrong for failing to strike up a conversation with this lively chap or was my course of action appropriate given the circumstances?

Have any of you ever had similar experiences before?