Because they are still family in the end. He spent years trying to win the favors. He believes he has no one other than his family that wants him without a reason. That's why he is so suspicious of ido too.
bc he still considers them his family, even if they mistreated him? sejin's kind and self-sacrificing, and everything he's done for 20 years is to try to be helpful to his family, it'd be weirder if he immediately revelled in their suffering
Is sejin dumb?? why would he be mad if his dad (who abuse him) is in prison or his family is suffering