Straight Guys that Rub your Back in public

I'm a guy. There's a male department head that I do work for that has a very annoying habit of rubbing my back, not quite in a gay way, but more as a gesture of encouragement or endearment. However, it is definitely the "slow hand" variety. He's married with kids; very smart and otherwise pleasant to be around. Sometimes it's my bicep while we're shaking hands. It's always in public and in plain sight of other people. He also does it to other men in his department and he seems to be comfortable with it. The others don't object but I'm guessing they're having the same thoughts as me. He did it again to me yesterday. I made up my mind that if he does it again, I'm calling a time-out, pull him aside and politely tell him I would prefer he not do that.

I know some people grow up in families that could be described as "huggers" or "touchers" and develop these habits in their formative years, but I think it's very inappropriate simply because 1) he doesn't have my permission and 2) it's a major invasion of my personal space. I also think this is a classic trait of con artists and/or a method of manipulation.

This ever happen to you? What do you think his motives are? What would you do?