Responses
I don't think anyone will be mad at you haha even the characters in the manga have repeatedly told him how selfish he is for searching for "Treasure" far away while leaving the most precious Treasure behind. To be fair though, he did promise to stay when they were in the hot spring-- him and Ayako hadn't been together that time yet (and he didn't have any inclinations towards her at that time) so he did plan to stay for Takara. Being with Ayako probably only solidified his reason to stay, but Ayako wasn't the original reason why.
I'm sorry I wanted to upvote
Just from my opinions, so don't be mad at me guys, I think kou-san was kind of selfish, from all this years he never thought of coming back to japan. But when ayako-san came to his life, he changed his mind. From my point Takara is just the sweetest for accepting it. If it was me i would be mad. (づ￣ ³￣)づ