Responses

It’s not exactly a lie. I think he means he hasn’t been with her ever since he met the FL’s mom. All the women in the harem he was legally married and been with, before he met the mc’s mom. Which is when he fell in love for the first time. It’s likely he didn’t do anything with any other woman after meeting the mc’s mom. (Hence why the FL is the youngest child and last)
He's telling a lie. He touched all he's wife that's why he has many children