I think the Shi Fu is her past life lover. As the Emperor, then samurai Soji, vampire king...

i kept thinking tht too, but shifu said tht its their punishment for doing bad stuff in their previous lives meaning he was also bad in his previous lives except those people you just stated were not bad, so i think its not him but different people i might be wrong tho

maybe what they did bad was make Siyin upset throughout their lives kekekkeke