"No, but everyone has to start sometime."
Not to be judgmental, but that is not a very positive answer to give. Consider answers like this next time:
"To be honest, I haven't had the title and official 'direct reports'. But my duties with previous employerX had me managing multiple tasks and delegating work. I don't mean to trivialize the task of managing resources, but I feel that I have had some strong mentors in this area and feel ready for that next step. I'm excellent at X, Y, Z and that should make the transition a little easier."
I know that's a lot more words than "You gotta start sometime.", but trust me, it will make a much better impression.