I think he started hating her BECAUSE of the accident.
No the bastard loved his family but after he lost his wife in the accident he was blinded by his grief of losing his wife and blamed it all on poor FL... probably
By the way why did he blame the fl when you clearly see the morther dying trying to save the brother cause it split in half in he fell out her hands?
Who knows what's inside that bastard's mind but he wasn't with them at that time so he probably just loved his wife and not his daughter since the brother is the heir and the bastard just probably saw the FL as a tool of marriage to make connections and gain more political power... that's just my theory
I think the father sabotage the carriage to kill the FL but things get sour the mother and child dies instead of the FL. So dickhead change the plan to "let's make her life a living hell"