
Given that brief flashback..... he was raped wasn't he?
That's why he doesn't like women, and doesn't want to associate with MC & Noah as they are triggers for his trauma.....
At least he didn't physical abuse the child and provided for his basic needs (somewhat) unlike in some stories....
Though I know that doesn't entirely excuse his neglect, it does make it understandable to an extent