Boat drifting problem?

I have a 18x54 flat bottom gator trax with a 45hp mud buddy. Boat is great for hunting and is set up for fishing. Trolling motor, live well, etc. the problem I am having is the back end of the boat just drifting in circles with the slightest bit of wind even with the motor in the water. I am fishing deep water so I can't use a power pole. Any suggestions on how to fix this problem?