I thought he said that since he was referring to the sex he had with the ex-gf. Like since he fell in-love with the white-haired guy's paintings, he had sex with the ex-gf who was basically impersonating him (i.e. acting like the paintings were her own). That's what I got from that. Also, him being a dick to hit him where it hurts.
Did they? When I saw the raws I thought that the crazy seme went a little crazy and when he saw the uke girlfriend he imagined the uke in her place, and that's why he had sex with her. He thought in his head that he was having sex with the uke. But I guess I'll have to wait and see. The manhwa is already at chapter 18 and they make some nice improvements in their relationship.
I don't fully understand why the crazy seme said "Do you still remember having sex with me?". The only sex scene so far in the manhwa happens in the near future, since the uke in that scene has the tattoo and the different hair style than the one in the college years/past timeline. So maybe they had sex in the past, or it was mistranslated or the seme is being/acting weird.
Overall I'm feeling really bad for the uke, only people that are either interested/using him for his painting or are bat shit crazy (in the case of the seme both), goes after him. That must be pretty hard not being loved for who you are. So I'm in the fence when it comes to the seme, if he is too crazy I don't want to ship them. If all he did was because he was trying to protect the seme from the toxic girlfriend maybe I can support him, but if he only get's hard or are interested in the uke because of the paintings that would be too weird.
PS: just looked at the raws and I think I know what happens and why he says that, just two chapters more before that