I'm not japanese but the reason for this is that sweets are usually presented as cut and fluffy and normally that's a girls things. So a man been in this stores looks unnatural. Since coffee shop are as well different there fluffy cute and stylish to fit the asthetic as well some of them even have rules that man are not permitted
I mean to an extent we (Americans) do it as well but with beverages. Sweet, colorful cocktails are considered "girly drinks". Black coffee is considered "manly," while pumpkin spice lattes are notoriously associated with women.
Like you might know assigning gender to random things is nonsense, but you're more struck by the absurdity when you see another country's silly made up gender rules from the outside
It's interesting to read the comments. I'm French, and at home it's not gendered at all. Everyone buys sweets, and pastries are just a cult ahah. Seeing a man not appreciate pastries, it'll be more weird for us than anything else xD. I suppose that in Japan, sweets are very connoted with girls, Valentine's Day and all the “kawai” aestheticism. Over here, it's more of an “art de vivre” for everyone.
Kinda random but I had never seen guys be embarassed of going to bakeries or sweet shops before I started reading manga. This thing that they say that desserts and sweets are girly and only women like them is so weird, so different from what I know lol
Maybe it’s a Japan thing idk