So what I imagine is that they are under a spell/curse that fades when they remember a certain time or feeling or when they are away from grace for a specific amount of time, I believe that the Duke is jealous because at some point he did love her till that spell manipulated him. Same goes for the other guys till they were cut off by grace calling them. So I imagine the Duke is annoyed because his heart knows his true feelings. I love angst so this will be yummy