
I'm wondering if in the original timeline if the old servants spread bad rumors about Reina, so the duke thought the situation with the baby was worse than it actually was. We know the old servants had no respect for her so they might have exaggerated some stories when they told the duke that she wasn't even touching the baby.
I'm not trying to excuse what he did, he clearly should have taken the time to find out what was really going on, but it might explain why he was so willing to remove his child from the situation.