I'm not well versed in divorce law, but, if she screwed up the marriage by cheating on you, then shouldn't it have been easy for you to get full custody of the kids?
You won't see many judges give full custody of the kids to one parent or the other for marital unfaithfulness. Just because his wife cheated on him doesn't mean the kids are in any danger in her presence.
To the OP: is there any "dependent care" you paid for the kid that was living with you? If you didn't send him to daycare, maybe you paid a relative to keep him? It's not as good as claiming the kid but at least you could lessen your tax burden some.