I believe its both. Assaulting him and being unfaithful. Though I don't blame Slen for being that way because of Toguto's mindset. But yeah, I'm more supportive of Alta and Illeweth's relationship than theirs.
Slen truly loved him, but Toguto's words hurted him dearly. That's why he hated him & couldn't forgive him, how he only sees him as Another White Deer in a political point of view. So their friendship meant nothing to Toguto that time. Slen started to change, but still deep down wishes for him to notice him from who he was. But Toguto never changed his mindset. That's why Slen was seducing & sleeping around with every potential mates for Toguto (Is probably his way to Rebel against him). But even by doing that, won't change anything. So guess he snapped that time & assaulted him? (Though we couldn't really see, if it really was or perhaps he didn't continue. You never know he might have stopped & run away that time. Because Toguto has mentioned seeing Slen's angry face). Then Toguto woke up from reminiscing the past in his dream.
So... Slen assaulted Toguto, right? Slen said he needed many partners, pushed Toguto down, Toguto said "stop," the panel faded to black, and Toguto says he hates Slen and will never forgive him for what he did that day. That's assault, right? Or is he talking about how he discovered on that day that Slen was being unfaithful? Either way, I'm much less supportive of them being in a relationship now. Give them both someone less toxic to be with.