Yeah, but since he's technically underage as well, I don't think it's as bad as if he were something like a 50 year old guy in a young boy's body fawning over little girls...you know, that sort of isekai.
This is one of the few redeeming qualities of the manga. The MC isn't a lolicon really. He's barely a teenager himself as well, and "lolicon" refers to older teens and adults liking much younger girls.
I'm going to stop you there, my friend. That is not how this works. Let me explain to you why adding the years to your first life onto the years of your second life doesn't make sense.
Let's say that you have a 10 year old who gets reincarnated as a 50 year old without any of the 50 year old's memories and that person wants to date a 10 year old in their new life, Would you be okay with them dating?
No! Of course you wouldn't. Because they would still be in the body of a 50 year old.
What I'm trying to imply here is that you are treating the spiritual age as if it is the the same as the physical age. It's not. It doesn't matter how many lives you have lived or how long you have lived in your past life. If you reincarnated as a 10 year old child then you are a 10 year old child!! Your spiritual age may not have changed BUT YOUR PHYSICAL AGE HAD RESET in that case. And it doesn't make sense to apply matters of the spirit to matters of the body.
...lolicon...