It was not just the fact that he was "simple and not all enough" even tho that alone shows his inferiority complex... But there is more... First he was not in love with Aster's mother... He had already another woman in his heart but he was so weak hearted that he couldn't protect her and their child (she was pregnant) from his mother... Which made them break up & have the girl abord the baby... After that he had a marriage of convinience with Aster's mother...His mother kept tormenting her and getting in their family matters but he never took sides and tried to "stay neutral" which I believe is enough reason to make her feel sufocated... The fact that Aster their own child hated how her mother was treated while growing up, to the point of using magic to make her go back in time and give her a 2nd chance says a lot too... We never got too much details about their marriage honestly... But it seems that he was an inactive husband who was not there when needed to his wife and daughter and who gave into his mother's words or at least never retaliated... Sure compared to other isekai Manhwa he is far from a "villain" but these traits he has, though not sharp enough they can end up killing a marriage slowly...
I've just started reading this, but not her husband catching strays for being simple and not tall enough
(￣∇￣")