I bet normally they would have more people covering it, but everyone can't stand guard. They were having a huge celebration for restoring power. I think there are also advantages to not having your armory closer. You don't want children stumbling in on it. They didn't have a need for it to be close until they met the apes anyway. They were still underestimating the intelligence of the apes the entire movie. Also, from all indications, apes had no interest in using guns until Koba shot that guy.
ETA: Also, the armory was hidden. The only reason Koba knew where it was is because he secretly followed some people to it. If there was an attack on the town, they would probably retreat to the armory.
This post was edited on 7/15 at 9:48 am