Any advice on staying in Portland vs Bar Harbor?

Are you just going one place and staying?

Portland is good, Bar Harbor is worth seeing. But the value in Bar Harbor after an hour or two is proximity to Acadia. If you're not doing that, it gets kind of cheesy fast, especially as the cruise ships come in, IMO.

Which isn't to say BH is unattractive - it looks cool, but Acadia is the show. Acadia is also accessible and less intimidating than most NPs. Personally, I wouldn't go back to BH unless I was going to do a good bit of time in Acadia and/or do a lot of exploring outside of BH.

That said, I also can't really recommend staying only in Portland, there are tons of cool towns on the way to BH - Camden, Boothbay, Rockland.

Is there multiple days worth of stuff to do in Acadia while lugging around an infant?

I leave one month from today. Wife's family wedding in Portland, then we are going to Acadia national park then up to Halifax.

Why not both. We did that and it was perfect. Both are great but different.

Portland is a real city and as mention leans left but whatever the food and drinks are good, the scenery is great and that what really matters.

Acadian is a super special National Park with a lot of short hikes like you mentioned. We stayed in Southwest Harbor but it was just a short ride to Bar Harbor. Its not a terribly big NP so I don't know if it matters once you are on the island.

Have fun and Enjoy