This artist draws him meaner than he actually is in this scene (he's absolutely deadpan 99.9% of the time). But Ferdinand isn't in the wrong. Their society is not built like ours at all. Everything from their history, sociology, phisiology, geography and economy is not comparable to our own at all. So it's a bit meaningless to judge it according to our standards.
Even though it's unethical to us, Roz is effectively stealing that town's source of income and labour. Ehrenfest is not a wealthy duchy, so everything the peasants do is to sustain their livelyhood for the day/month/year. Moreover, to nobles, peasants are somewhat parasitic in that they don't contribute to the sustainment of the land.
To make a comparison. Say you owned a business that hired troubled teens, sure you have adults to run the store but teens are your main working force. Now imagine if some rich person came in and said they will bring those teens to a refuge center with better living conditions and an allowance. And if you didn't hand them over they would call child services on you. Now you have no workers and your reputation has plummeted.
Ferdinand is getting more and more repugnant to me when he act like that