Also who he believed her mother to be was his first wife whom he cared for (but wouldn't protect from 2nd wife family) plays a big role.
Whereas, Keira is the spitting image of the 2nd wife he was basically forced into marrying due to her family's political pressure. The same family that ruined his marriage and the life of his first wife.
This becomes obvious when you observe how he interacts and treats the girls.
In the first timeline he knew Keira was his daughter but went with the lie that she was a child product of affair the 2nd wife had.
He also played an active role in her execution by voting YES to have Keira executed when a ballot was done to decide her fate.
The duke is the only dad in a reincarnation manhwa I've ever hated(him and the other duke from "I tamed a tyrant and ran away tho atleast he regretted it) I saw on a spoiler that he only ever accepted corset bitch was so that the nobles would still respect him