Sensuality is the keyword i think. Anything can be sexual if you put your mind to it, the fact that females have hips and boobs or anything at all feminine is kinda what nature wants, while males just tries to find someone to mate with in general. Its however true that tribes in the past didn't care if women were topless, since it was natural. But shaming as a cultural norm likely developed this. I am not really sure honestly, i just think most likely that men is masculine so you can't shame him for being topless.
But good question i'll give you that.