Bad contractor advice.

Tldr: contractor did the job, asking for more money afterwards. Is this dude just taking me for a ride? What should I do? Should I nut up, and tell him it's his mistake, he needs to eat it , or split the difference of $800 ($400)to get him out my hair?

So I'm getting my drywall done after the flood. I contacted a Sheetrock contractor with many crews, off of a referral. He said he could get on it as soon as I got my sheet rock, so that was a huge incentive to me as I've had multiple people come out and not come back even after giving me a bid.

On our first conversation he said he would do 50 sheets for $2200. I said "OK sounds good, do you want to do a contract, or come out to look at the job?" He said "no he's been doing is 10 years, and in the last two months he hasn't done any contracts". I expressed my concern about him after the job being finished asking for money, and he said "no, that won't be a problem."

He sends the crew out to hang, and they can't speak a lick of English. But they were very upset once they got there because it was more work than they had been told, Because the owner had not been out there to see for himself.

So I talk to him again, and he said that it's now going to be $3900, and $2200 must've been a mistake, He had no idea where the number came from and this crew wants $1700 to finish the demo and hang the rock which ended up being about 35 sheets
That's still not a bad deal, and they were there, so I said all right let's do it. He ended up coming out to give me a better estimate on the finishing part, bid was $1900.

The finishers comes out, and I show them around the house, and the things that I was looking for. Two hours later he leaves, and the next morning a different crew shows up for reasons unknown and I explain the same things to them.

I go out of town, but still am available if needed. The owner calls me when the job is done, and tells me they did a lot more work than expected, and that I wanted the job done like a custom home, and that I asked them to do work that him and I didn't talk about.

He then suggested that I pay him 800 more dollars!

I told him that's not how it works, and that I barely spoke to the crew when they came out. He said look, you can either give me the extra money or not, it's up to you it's your conscience.

Is this dude just taking me for a ride? What should I do? Should I nut up, and tell him to F off, or split the difference for $400 to get him out my hair?

It seems as if he's a crappy contractor with horrible business practices. It's his mistake for not communicating with the crews, and making sure that we're all on the same page.
I wonder if he does this often, and passes his mistakes on to his customers. I also feel partly to blame because I did not get things written in contract, but I'm pretty sure even if I did he would still ask for more money.