What Are My Chances of Getting This Job Now?

I am attempting to transfer positions in my current company. HR treats this like any other job opening open to both external and internal candidates.
I had my interview on Monday and did well. He said I should hear something by Wed or Thurs. I hadn't heard anything by yesterday and contacted HR. She said she hadnt heard from manager on his decision either and would contact him. Still nothing by this morning so I contacted HR again and she told me he'd like me and another candidate to interview with someone else before they make the decision. They suggested Monday. Here's the problem: I'm on vacation all next week. I would postpone even but everything is paid for. They said no problem and suggested the following Monday. I'm afraid that my unavailability is going to cause me to lose out on this position that I really want though. I am highly qualified for this but I feel like they want to fill it asap. Am I right to worry?