I would have not let him go. You noticed he was tense and then he said "no" when you said you were leaving. READ THE SIGNS BRO!!
