It started after - well, Ianto generally referred to it as the Zagreus business, for lack of a more socially acceptable term. That time Zagreus broke into my mind. The Doctor picked up Fitz and Ianto picked up the hobby of strengthening his mental barriers.
He found a few books about it in the library and set to practicing with those; building the imaginary brick walls, reciting the nursery rhymes, that sort of thing. It was like his training at Torchwood One all over again (but this time, he wasn't being evaluated).
The day came, of course, when he learned all he could from the books. In theory, he ought to be a master at it, but without really testing the barriers he'd never be certain they worked. He finally asked the Doctor awkwardly over tea (over his tea, anyway, as the Doctor was doing something underneath the console). So... can you do that thing, y'know...
The Doctor had been surprisingly supportive of the idea, if somewhat wary of his own involvement. He promised, with a smile that Ianto could never quite pin down as joking or not, that he would go easy on Ianto to start off.
Still, it was like being hit in the head with a bowling ball. All of those stupid books for nothing, he thought, but the Doctor assured him that they could start slower, work up to it, that Ianto was already leagues ahead of most of his species (which managed to be a compliment and an insult).
They met when Ianto felt like it and occasionally when he didn't, because the bad guys won't wait around for you to have a good day, after all. It was embarrassingly tiresome for Ianto, but after a month or two at it he'd improved enough to be able to keep the Doctor at the metaphorical door - not enough to push him out, but enough to keep him from coming any further for a little while.
Which, he decided in the end, was useless. You're holding back, he'd accused. I can't push any more if you don't give me anything to push against. He was too frustrated to keep at it, that day, so they decided to give it a week. Ianto would have week with his books and his brick walls and his nursery rhymes and then they would try again - really try.