I think it's fair to say that his cognitive function dwindled after the accident and head trauma, it goes along with the retrograde amnesia he suffered, he might also experiencing slight and barely noticeable anterograde amnesia. This caused him difficult to process new informations or understand the input itself, as it is seen in some chapters where it's hard for him to understand things being said to him, despite it wasn't anything hard to understand. Meanwhile it could be said that his cognitive issues were due to organic cause (head trauma), seeing as it never was said clearly whether a part of his brain has shrunk and his cognitive function relates to his stress (he gets better in the ending as he's happier), it might also be due to mental issues which caused it (perhaps in dissociative disorders).
So it's understandable that his cognitive function ("intelligence") come back gradually as his brain and environments are in better situations now, although for the memory part is a bit tricky, it's 50:50 for it to come back or not
I think his intelligence has shrunk a little and he never tried any treatment so it stayed that way. Only when he started to stimulate his brain more and was placed in a better situation (without his asshole abusing brother) did he get better in terms of intelligence.
I guess if you lost your memories and you were told you're stupid you'll believe that you are. That's why I think in that chapter where he was asking for advanced sodoku, it's saying that he wasn't stupid at all. People around him treated him as one, that's why he also thinks he is and eventually really made him into one lol I still can't forget the brain part. (╥﹏╥)