Okay, I don't know what chapter you gave up on or if you continued after you left this comment, but these are explained fairly early on if I remember right. Spoiler warning for obvious reasons.
Technically spealing, the oracle isn't fake. MC is biologically 100% the Duke and his wife's child, but here's the thing - due to a ploy, one of the royal family members (Lucas) is stuck inside the Duke's body, able to occasionally possess it. It's because of this that MC was able to inherit magic powers from him, which makes her his child in technicality.
It wasn't the Duke who started avoiding MC nor was it bc of the news, it was just unfortunate timing that Lucas ended up possessing him/gaining control of the body when all of these rumours/news spread. The MC recognises that her dad wasn't the one in control of his body when she sees him.
The Duke doesn't remember the times when Lucas is in control, I forget how aware of it he is. I just remember that the Duchess had also recognised the switch and knew the truth. I do know that the Duke loves all of his children.
I see
I feel like this is all someone's plan or something, likely the emperor. Fake news/oracle to break the family or you know, the typical thing so, nah, I had high expectations but it's frustrating. And you can't continue to love the child whom you've loved for three years just because she doesn't have your blood? That's pathetic.