Authorize.net and credit card processing question

I have been using authorize.net as a "payment gateway" to accept credit card purchases for a website I run. In the last two months, I have had 3 transactions (all American Express) disputed and American Express has refunded the customers' money by taking it out of my account. Authorize.net claims that they are not responsible for these transactions and verifying the credit card user's info is my responsibility.

Does anyone have any experience with these types of disputes? Who is at fault here? Please let me know if I am missing any pertinent info.