I am in no way defending the pos of a father but unfortunately because of the empress' family having a high position themselves he cant just straight on kill the empress for his own selfish reasons. If he does that it might uno reverse and he could get overthrown (which I dont mind) and seeing how the empress' family is I dont think they'd make great rulers. The only thing I dont understand is how the emperor hasnt collected any proof of the empress' and her family's wrongdoings. This would have made everything easier and could have solved everything. Also I dont remember everything from the start of the manga and Im just going off what I remember so ( ̄∇ ̄")
I dislike the emperor more than the empress, at least she isn't a hypocrite in her aim for power. He's just a coward who can't even properly try to fight her head on.