Oil makes latex condoms degrade, so it's not good if you want safe sex.
With women using babyoils and such comes with a higher risk of yeast infections and such than if you use water-based lube. Generelly oils can fuck with the pH and environment in the vagina.
Not entirely sure what effects oils have anally (besides destroying condom).
Opps, totally missed this till now.
Exactly what Raphael said. Also, Baby oil is made from petroleum (same thing used to make gasoline, kerosene, etc), not a good chemical to have internally. If used anally it can be absorbed through the lining of the colon and it is difficult to fully remove off the skin. So having used it once or so isn't the end of the world, but it shouldn't be used on a regular basis.
Do not use baby oil for anal or vaginal sex kids.