ETa: On a similiar note, I believe in that chapter Theon sees Whoresbane and Manderly chatting which is very interesting. In ACOK, Luwin and Rodrik commaned the Umbers, Whoresbane and Mors, to work with Manderly on water defense against the wildings. That would lead me to believe they've been in contact for a while and are in league against Bolton.
Yup. Makes you wonder what Manderly plans on doing with his hidden fleet too.
Also, just about every noble family in the north has reason to turn against the Boltons/Freys, whether they had someone die or get captured in the Red Wedding, Sack of Winterfell, Roose's Duskendale trap, or Ramsey being a douche bag.
IIRC, other men (besides the men you mentioned) that are in Winterfell with Bolton are:
Cerwyns - Lord Cerwyn's son died in the Sack of Winterfell
Hornwood Men - Lady Hornwood (who's also a Manderly) was imprisoned and starved to death by Ramsay
Lockes - The Lockes lost men in the Red Wedding. Robett Glover's wife is also a Locke (Robett was also one of the men sent to Duskendale by Roose, but was traded for another prisoner and ended up in White Harbor).
Tallharts - A Tallhart died in Duskendale and another in the Sack of Winterfell.
Also, aside from Mors' men outside Winterfell, I think there may be another army waiting somewhere from White Harbor. Other tidings were of greater interest. Robett Glover was in the city and had been trying to raise men, with little success. Lord Manderly had turned a deaf ear to his pleas. White Harbor was weary of war, he was reported to have said(adwd, p. 198).
Manderly weary of war? We know he's been plotting behind Roose's back, so I doubt that. Plus, Manderly tells this to Davos: “I have been building warships for more than a year. Some you saw, but there are as many more hidden up the White Knife. Even with the losses I have suffered, I still command more heavy horse than any other lord north of the Neck. My walls are strong, and my vaults are full of silver. Oldcastle and Widow’s Watch will take their lead from me. My bannermen include a dozen petty lords and a hundred landed knights. I can deliver King Stannis the allegiance of all the lands east of the White Knife, from Widow’s Watch and Ramsgate to the Sheepshead Hills and the headwaters of the Broken Branch. All this I pledge to do if you will meet my price.” (adwd, p. 392)
The Boltons and Freys don't stand a chance.