Maybe he has always liked her? They have amicable relationship before the brother stepped in. He began to resent her, but when he lost her, he realised that he actually liked her. That's my theory anyway.
Feelings are not all that... for all he knew, SHE DESTROYED HIS FAMILY AND HATES HIM. Love can never bloom in such a toxic environment, I mean, he was trying to kill her before her reincarnation that is before she killed her.
I think it's just the Author taking liberties
But we don't know what he knew? And the fact was she wasn't the one destroying his family. And he wasn't trying to kill her. He was trying to kill the king and that's why FL stepped in because she didn't want the curse to befall him.
... I would love to know when exactly he fell for her?? was it when she destabilized his Duchy or after she facilitated the death of his parents and sister albeit indirectly...