There's some affection too
Hmm, it has changed, I think. If you notice, yeonjo mostly did it to stop the killings, much less because of hate on killing his family.
He's mad at HIMSELF for getting soft and letting his guard down because he admits himself that he was suspicious of Yeonjo from the beginning, somewhere along the line he lost sight of that and allowed Yeonjo a lot of freedoms and leeway, which gave him access to the possibility of acquiring and using poison.
Uhm duh? ಠ_ಠ What do you expect? Yeonjo might have gotten distracted from his plan but that doesn't mean he hates you less. You killed his whole family ಠ益ಠ