That's what I'm thinking. Hoping actually. I want to see flashbacks from his time on the island.
If nothing else, he's a billionaire before the crash, billionaires do crazy things with their money.. martial arts is one of those things. It could very well be that he was already good at martial arts before he was trapped on the island, and because of the island putting him in the best shape of his life.. he can now do that much more, simply because of his physical potential.
He would have had to of hunted etc. on the island, so moving quickly - over mountainous regions etc. would have helped him with his fast and quiet movement.. If they go away from someone else being on the island, I would just move towards him having been trained before the island.
That being said, his bow looks special made, and certain things he did in his "lair," I don't think he was completely cut off from all human civilization on the island.. something happened on that island. (maybe that island was very popular, but he ended up having to kill a bunch of people who tried to take him from the island by force?)