Maybe it's a matter of respecting his privacy, at least the one in his skull.
i think gorou is rejecting verde in his head too. top right panel:
http://www.mangago.zone/read-manga/kimi_to_kore_kara/bt/312313/Vol1_Ch4/18/
we the readers knows the reason behind gorou's imagination. but what if that exact panel pops in the forefront of gorou's mind right at that moment. it showed imagination!verde trying to get close to gorou and imagination!gorou horrified reaction. then followed by gorou's pushing verde away. verde misunderstood and kinda froze, maybe shocked/blanked out and didn't read gorou's mind further. that's just my interpretation tho'.
of course there could be some other reasons. or it could be just a plot hole where the mangaka forgot about that mind reading power.
Shouldn't Verde be capable of reading Gourou's thoughts?