I think the problem started with how Suzuharu's dad raised him. He's more afraid of his dad more than anything it seems. He was even willing to throw away his love :'((