It’s not exactly a lie. I think he means he hasn’t been with her ever since he met the FL’s mom. All the women in the harem he was legally married and been with, before he met the mc’s mom. Which is when he fell in love for the first time. It’s likely he didn’t do anything with any other woman after meeting the mc’s mom. (Hence why the FL is the youngest child and last)
They got together again years later. They met outside the palace and stayed in a small house and fell in love. She didn’t know anything about his past until he wanted to take him with her to the palace. That’s when she found out that he was the human emperor and was shocked. She didn’t want to be another concubine fighting with other women, so she ran away back to her kingdom. The FL’s mom was the only woman he ever loved, the rest of his concubines/empress and children were just political marriages which happened before he even met her. She doesn’t like the merman
He's telling a lie. He touched all he's wife that's why he has many children