The way I feel when he said that line was to show others that he is his, while not exposing their romantic feelings. Like hinting subtly that he is his lover and not to be shared but he can't really say that out loud.
Don't get me wrong, I fully understand that that was supposed to be a romantic moment. It was just badly executed one. If he said "He's one of my people." I'd live for it, since it would clearly show that he sees Dev at an equal level. When I read that, i don't see "despite their social standing, they've found love and comfort within each other", but "Rich dude, bought a slave and is now possessive of it.".
Yup definitely would have been better if he said that instead I see it now with the last part you said
I feel like the story should be better if the spinels had any other distinguishing feature beside dark skin, hair and eyes. It's just uncomfortable reading a story which deals with a real problem (slavery), without an ounce of realism. When the duke said "because he's mine" or something, i vomited in my mouth. Maybe if the duke tried to get into government and change the laws or some other magic solution, I'd be fine with it, but the dude does the bear minimum of not being a total asshole and is treated as though he's an angel from above. Can't tell if this is some weird savior fantasy of saving a black slave ( but don't worry, he has straight hair and blue eyes, so you don't have to put yourself down), or if this is just incompetent and not at all thought out storytelling.