I feel like that’s pretty harsh. He was a very young father and it’s not like he knew the current empress was gunning for the former. What was he supposed to do never let the former empress leave his sight? Also, he kept investigating for all the years, but he had very little to go on and inaccurate information. Also, he’s not entirely powerless. He’s certainly managed to protect his son the entire time. If he recognizes and brings her under his protection from his point of view it maybe safer than leaving her defenseless with the current empress gunning for her. Remember he doesn’t know the current empress was stealing her power and that the duke is aware. In a sense he lost his wife by letting her leave his side he may feel she is in danger
The emperor knew there was a foul play with the late empress death, did he even launch any investigation? Is he so powerless against the current empress? He can't even confirm the baby's gender until recently and how she's been living until now. It's like forcing her into the lion's den that is the imperial palace and pushing her close to her deathtrap. He's useless both as an emperor and father.