I mean it makes a lot of sense
It's very common with narcissistic parents to isolate one child and point all of their abuse towards them. In fact, such behaviour only makes the isolation more effective.
Also, in this very specific case, all the other children were raised under his supervision and care. So it's only natural that they are assholes just like him, and they get along well.
This is fiction. In real life, even if he'd have miraculously noticed Tae-moon's misery, he would've blamed Tae-moon for it. He would be like "it's your fault that you were a bad child and I couldn't give you the love and respect you needed" or maybe "you should've been able to overcome that already. You suffered because you're weak"
It's surreal how long it took him to actual SEE taemoon and see his misery, like come on, u have other children too so don't u realize how ur actions have affected them as well as their lives, like fighting each other for inheritance, etc. Even if it took him a while, he did realize eventually that he's pretty much been an ass, hopefully there'll be a development, but we'll see.