re: If I'm looking to make a 5% annual return (after tax) by investing in funds...

You're going to have to take on significant risk to make 4-6%. Not too long ago, zero risk money market funds were paying 5%. These days they're paying basically zero. CDs and lower risk bond funds will get you 2-3%. To get above that, you're going to be generally holding stocks. (Assuming you're not into other things like real estate, etc)

The answer to your specific question of investing vs paying down mortgage is a matter of risk tolerance, among other things. Can you handle a relatively short term drop in value of 20% or more? Do you have other savings or investments?