He's still so oblivious to his feelings. He's obsessed with the wrong lady. Why would anyone follow in secret someone who was a "playmate" for a noble? If he just stopped and think a little harder he'd come to the realisation that he was an asshole to the wrong lady.