Because they are the head family and first collateral line and he is the third branch, but I think he feels self conscious that he doesn’t have a guardian yet like he isn’t fully an heir since he doesn’t have a guardian yet. I also don’t think he wants to be pitied.
Eh ... why is Chen so conscious of these 4 ? They're not the type to be bullies so maybe it's because he doesn't have his guardian ?