I just think that Varys has figured out that he has no interest in the squabbles of King's Landing. And if the theories about Aegon and Varys being Blackfyres is true, then really Varys and Littlefinger are more or less on the same page on what they want to get done. Varys' plan is just much larger, grander, and more patient and underground than Littlefinger's is. Littlefinger is a player, but a player that the even bigger one has accounted for.
This post makes me think that we're running in circles around each other. Both trying to say the same thing but not quite getting each others point.
I do think littlefinger is a very legitimate player in the game and, using the college football analogy again, has the talent and potential to ruin someone else's season but he just doesn't have quite enough to win it all.
The true powerhouse, the true master manipulator and the true master of the board (Varys) has already accounted for any moves and trickerations that littlefinger and/or Tyrion can bring to the table.
The difference in littlefinger and Tyrion is that littlefinger's presence leads to his company hating him. Littlefinger has no friends everyone of his pawns KNOWS that they are his pawns and that doesn't win him any favors.
Now let's look at Tyrion. He puts off an air that forces people to like him. Jon Snow, for instance, hated his guts before they left winterfell. Yet before Tyrion left the wall Jon shook his hand and called him friend. Bronn met Tyrion while Tyrion was a prisoner and Bronn was his guard yet Bronn became his closest companion. He talked his way out of slavery and into the most powerful sell sword company not named The Golden Company. He wormed his way into friendship with Jon Connington to the point that Jon risked and ultimately contracted grey scale solely to save his life.
Those are the reasons that Tyrion is the most powerful man (not counting Varys/Illyrio) in the series to this point. People have no choice but to love him and that gift is why he is still alive.