He probably had a hard time losing his wife and such but he still could've manned up to take care of her during that time or just ask for visitation with Yui's uncle but not straight up like he did. He's selfish for not asking Yui before he took her in the trio and spoken about it with the whole family but instead takes her away and decides to tell her. Children know a lot and can understand more than adults realize.
Ha Iove it. The evil cackle of a proud teacher