Skinny people can't diet

I started dieting (not to lose weight, but to gain muscle but keep off fat) a few months ago, so I try not to eat foods with a lot of carbs, avoid fried stuff and sweets, and consume more protein. My family loves those types of foods and so do I, but whenever they buy them, I refuse to cave in to my cravings. Every time my mother sees me doing this she keeps saying that I don't need a diet because I am slim already. I've been told this by my friend once or twice as well.
Has anyone else experienced this? Why can't I want to try eating healthier and exercise if I'm not skinny? It is already hard enough to try to not eat those kinds of foods, but I am also being nagged when I refuse to.