Well, Tatsumi never actually showed sexual or even romantic attraction to Subaru when he was a child... but as time passed he began to understand the anticipation he feels for Subaru to come back and he understood that this unreasonable longing that has last for so long is love.
But, after this realisation, he still didn't fantasise about the child Subaru he had in mind, and he was fully aware that he has grown up and became an adult.
Eventhough, it is briefly shown in the manga, these two have been in touch for years by emails and stuff... so the emotions haven't grown only because Subaru was an important presence in Tatsumi's life in the past. It is more likely that it has developed over the course of years.
So even if the story starts a bit weird, i liked it and it wasn't off-putting for me. It is not pedophilia as many people have mentioned it, cause that term has specific things associated with it that we don't see in Tatsumi's character.