Credit Card Question

I need some advice on how the following situation will affect my credit score, sorry if this has already been covered before.
While checking my card activity for fraudulent charges I noticed that I was charged a $15 annual fee, so I went to the bank and asked one of the reps there about it. She said that card had always had a fee associated with it, which I disproved by showing her by account history for the past 3 years, none of which had an annual fee charged. I was then basically told that there was nothing they could do about it short of signing me up for a new card, and voiding my old card. I've had the card for 11 years, I got it straight out of high school and have never missed a payment. What are my options and how will they affect my credit?
I'm tempted to cancel the card, the interest rate is not the greatest, no rewards, and with an annual fee now I can do much better, I am also a little put out by the way my query was handled. However, besides a few short term loans (3 - all paid off within a year), a mortgage and a car note, my credit history IS my card. How will this affect my credit score if I close out this card and open a new one with another bank? I've often heard that your score heavily depends on your longest line of credit, besides this card that would be my mortgage I started 5 years ago. Just a point of reference my average credit score was 790 last I checked a few years ago.
Or am I just over reacting to all of this and should just suck it up and pay $15 a year? I just think it's BS that all of the sudden I have to pay for a card I've had for 11 years. TIA, sorry for the long post.