Just a theory, but maybe he was reincarnated as well?
Or this early he actually did like her and only started to despise her later because she was a selfish little B*tch.
I have two possible explanations for his reaction. First, Elise mentioned that she was a horrible person in her past life and did a lot of bad things, so maybe the prince actually liked her from the beginning until he found out what kind of person she was? He might just appear to be a cold person, but that doesn't mean he didn't have feelings for her which he kept secret (sort of like a Tsundere?). So maybe he only started hating her when he found out she was a really bad person. Now, in her new life, it's like we're back to the prince's first feelings for her before he ended up hating her.
My second possible explanation is that the prince knows he has to marry someone somewhere in the future. So why oppose the marriage between him and Elise, if he's going to have to marry someone eventually anyway? Until now, Elise hasn't done anything harmful to him or the kingdom. She's also from a very influential family, so the prince hasn't gotten any reason to hate her or their union. In other words: he accepted the union, because it seemed like a good/decent outcome.
I didn't really understands what the prince is thinking. he said he isn't being forced to marry her, is he having some feelings for her? then why he treated Elise so coldly in the past. I can only assume, the reason Elise became the wicked empress because of her loneliness, she is already spoiled brat who can get anything she want just by throwing tantrum, when the person she truly love didn't even glance at her no wonder she turn out that why.