The second you use the word "man-child" as an ad hominem insult, then you have no credibility in your word, and I see no reason to see you as anything more than both a coward and a bully.
Chances are that if you use the insult man-child then they are talking to a mirror. Because guess what, it's a childish insult. It's the equivalent of maturity as calling someone a doodiehead
If someone is lazy, stupid or childish. Man-child seems to be the perfect word to describe them. However, if someone is called that for liking something a kid likes, yet acts mature and calm; its just another dumb insult like you said.