This is his second life not past life. First life he got rapped by his brother. Next life I believe that the brother just like the (main love interest) he was reincarnated like the main character. His feelings for his brother must have been feeling the same as his past life and he felt a longing towards him, but the MC hates him for what he did in his past life resulting in rape again from the brother. The father figures out resulting in, I believe the MC being sent to the camps to basically die. Hope that makes since.
in the present time line. so when you read it think of it as if the MC just got reincarnated and hates his brother and the main love interest for what they did to him in his first past life. So we are looking at the present time when he comes back but looking at the (second life past when he reincarnated)
What we last saw is part of his present life, but it did happen before the emperor got him from the slave area.
In his past life, his brother was older, and now the brother is the younger sibling, he still hurt the main character (blue eyed). He is still in love with his brother, which the main character hates, that's why as soon as he saw him in his second life he was cold and harsh with his words. He was forced again to endure his brother's presence, but the servant said it was main character (blue eyes) who raped the brother even though it's the other way around since they believe that the main character is the reincarnation of a mad person. While he is the reincarnation, he is not mad, he was abused, but that was something only the palace people knew (the eunuch explained this earlier). So instead of being known as someone who was driven to suicide by his pain, they painted his off as a mad man, which is wrong, and the reason he was locked away in his new life. It's sort of like his family feared him already and we're looking for an excuse to get rid of him.
Therefore, his present life goes like this: he is born with blue eyes (just like his past life), he gets locked away around his five years of age, then he gets his first visit from his younger brother (who was the older brother in the past life), he continues to visit and it's what we last saw on the chapter, the servant heard them, thinks that blue eyes raped his younger brother, reports to the fam, the fam sends blue eyes to slave work so they can get rid of him like that since a lot of people die in slave work, then people report he is from one of the important families, they think he is a spy (which also happened in his past life), he is taken to the emperor, and well as we saw the emperor didn't allowed him to leave and they almost repeated their last life.
I really hope I managed to explain this properly, English is not my main language you see, don't be afraid to ask for more clarifications, it would be easier if I could post photos and be able to point out certain scenes but sadly I can't.
Did I miss something???
I am so confusion