Vegeta never showed affection, aside from a few moments. Like attacking Cell after Trunks was killed, tipping his fingers as if he was waving goodbye to Trunks as he left in the time machine, or when he hugged Trunks, and had Piccolo take him and Goten away, so he could try and kill Buu by blowing himself up. You could also argue Goku had these same types of moments. Spending the days leading up to the Cell games with Gohan, always playing with him when Earth was peaceful, and taking Cell away with the Instant Transmission to protect the Earth. This discussion could really go a long way.
During the period between Cell and Buu, it is difficult to say how much attention that Vegeta gave to Trunks. I would think that most of his time he was obsessed with training. It could be that he didn't spend much more time with Trunks than Goku spent with Goten or Gohan, even with Goku not even being on Earth.
Wasn't it shown that young Trunks spent some time training with Vegeta? That would give the edge slightly to Vegeta. It could be argued that Vegeta spent more time toughening up Trunks for fighting than Goku, but Goku spent more warm, family-type moments with Gohan but not Goten.