
I think when he says "killing all those innocent people" he's referring to others the flame emperor may have killed in the tower as well on his clime to power. We can see that he would have killed gong-ja to keep quiet about what he's like; both in this and a previous scene, this means he's probably done it many times in the future without others realizing it.

Yeah plus Gonja's visible envy which probably was the largest part of his hatred, the presumed part of fe possibly killing a lot more people in the tower could've been solidified or canon if the fe mentioned it at least, but there was no mention of him being a cold blooded killer in the tower or having killed anybody else actually (when gonja met him first atleast) so I thought it was so weird lolol but it's just a thought, fe was portrayed a heartless scum later on (honestly more like a sociopath than a cold-blooded killer, or js both)

It's been awhile since I ready this manhwa but I'm pretty sure it was inferred that the sword saint was killed by the flame emperor. I mean, given the flame emperor's ability and the sword saint's personality (and his disappearance in the first time line that readers are introduced to), the flame emperor definitely killed the sword saint.
Chapter 4 in and the "he killed all those people" statement from the mc about the flame emperor had me going "????", like what??? wdym, the narrative had only shown the flame emperor killing the saint and then him, I just thought it was a funny thing lolol by what the mc has seen there's not enough to tell flame emperor is serial murderer, for one the saint did kill fe and it was self defense and his murder on mc was the scum part but not equal to "killed a lot of people". It's prolly just mc jealousy (incase somebody refute, don't bother it's just a dumb observation, I'll read it further and see What happens, no spoiler pls)