re: I have the disquieting feeling that the GoT TV series is starting to drift

Ramsey had Reek tell his dad in the scene where he was being shaved in season 4. That's why his father sent his the guy who cut Jamie's hand off to Castle Black to try to buddy up with Jon Snow and find the brothers. He found Bran in the hut, but Bran took control of Hodor and snapped his neck. Later in season 4, when Ramsey was given the name Bolton, he again asked his father about the other two Stark boys, to which his father said they didn't matter anymore.