My suspicion is that the Duke did/does love her in a tsundre/yandare kind of way. Basically he didn’t know how to actually express himself and so acted cold towards Viola. But now that she has “amnesia”, he’s taking the chance to be more expressive, much like she is.
I mention the yandare, 1) because of the creepy writing, and 2) I suspect that he did something to drive her male friend away a year ago, so that he could keep Viola to himself.
When I Faked Amnesia to Break off My Engagement, My Fiancé Casually Told Me a Ridiculous Lie -