So in Asimov world, what happened to the Spacers?

Maybe a better way to put it is that they became more concerned with personal comfort than advancing humanity. The most extreme spacers lived on huge properties alone and had severe phobias of even seeing people. They only met for the uncomfortable (to them) necessity of breeding. This attitude of personal comfort having the most value isn't conducive to colonizing the galaxy. You have to risk discomfort, suffering, and death to be a pioneer.