See I think the issue is actually that all of his lovers previously were selfish and dismissive, especially in bed. But I don't think the MC has made that connection yet and lacks the tools and terminology to properly convey his emotions (I really like this because it makes him realistic and relatable). Therefore it results in the belief that society consistently pushes and that's big d*ck= good sex, small d*ck = bad sex. It's a super common misconception a lot of people have, especially younger people who have yet to learn how to properly express their wants, needs and emotions. Though there are times that size can make a difference, overall it often comes down to technique and empathy when it comes to matters in the bedroom.
I mean ... the other guy left his long-time partner because "he had a short dick". And he chased after this one for the length of, well, dick.
... can't expect much from either of them.