That would be a good elk round, or stag. But IMHO it's way too heavy for southern whitetails. Not to mention the beating you take on a bench.
I shoot a 165gr Shift Scirocco 2 in my 30-06 loads. It sits atop around ~62 grains of Superformance powder. I think it leaves the muzzle around 2700-2800 fps.
I agree that the 180 is a bit heavy. My dad took some 300 Win Mag handloads to Montana with 180gr Barnes TTSX rounds in them. It dropped an elk and mule deer like a sack of potatoes, but I still think it would be a bit heavy for whitetail. I agree with Faxis that there's a lot of wasted energy.
Then again, if you're having success, don't change a thing. If it works, it works.