How hard is it NOT to pull on someone's jersey?

Watching Buff vs. NE. Pats had them stopped, but Pats DB pulled on receiver's jersey (which probably had no bearing on receiver's ability to catch the ball), but the ref saw it and called PI.

Teams should started fining guys for these infractions.