Wasn't he the rightful heir to the throne anyway?

Yes but you know how it goes in tgese: he probably didn't have much power (meaning nobles and followers) backing him like Yuri did which would be his mother and whoever else is backing her. He was powerless, which is why it was a good idea to stay away. If he wants to revolt and do a coup d'etat, he's still gonna need a strong and powerful backing which would be the Vadels and maybe their Knights and others to help fight.