He loved "Yunjae". His afecction was always directed toward him but he mistake yunjae for the glasses guy since they have the same name. You can say that he haven't be able to fully commit to someone else because of his childhood love until he met Candy, who "coincidentally" is the real Yunjae XD
He had affection on childhood Yunjae but the whole fall inlove thing is directed to the glasses guy not childhood nor present candy. That affection can't turn to love just on memory w/o the present, glasses guy, affecting him. He's "able" to commit, he just didn't act on it for a long time to glasses yunjae. Candy was very forward on 'wanting' him that helped them progress more but he's still fond of glasses guy╹◡╹
Now that's confusing because you can't say that he loves candy since childhood since the blooming of his lingering affection has been directed on the glasses guy along the way so it shouldn't change anything(?) It still feels like shit for me T◡T