There is a difference between remembering all the details of his lives and just remembering bits and pieces. Just like us, we read thousands of books, but we won't remember every details and each words. That what he did, he put a limitation on what he can remember so it won't put a burden on himself.
I mean he is aware of the past memories, but it is not him anymore, just some distant past totally unrelated to him now. Like reading a book, knowing something and experiencing it is different. Just because you read a book about a man living his life in vast knowledge, it doesn't mean you suddenly get all of his knowledge. You won't even remember much after putting the book down in 2 years, let alone that many lifetimes.
So he is free from the emotional baggages (that makes people mature and wise) of the previous life, and can grow as normal being (whatever being his current life is), even then some forms of feelings can be carried over, but not too much.
He remember hating a dragon because he was killed as a slime, but he won't remember the fear of dying, or losing his previous family as the elf, etc. The intensity is different.
i love the fact that he's aware of his past life but doesn't act like an adult in his current life