Since the ML is a concubine son I think he isnt planning to sleep with his concubines only with his values empress. The concubines are purely for the empire's use and hostage. He doesn't want a repeat of his childhood story.