Rising Dusk

Hi there, my ascending friend!  I'm good, thanks. You?
Those who supply men’s hats are called hatters while those who supply women’s hats are called milliners.

@Samurai Equine Doing good mine friend 

I wonder why the difference 

@Rising Dusk

There are a number of reasons, surprisingly! To put it simply, it came from a foreign concept of people Milan, where fashion designers (milliners) specialized in clothing and accessories for females, including hats. Later, it just came to be associated specifically for making hats for females. By a more modern standard, the difference can mean a number of things. A hatter be someone who makes simple, practical-use hats while a milliner makes hats for the sake of fashion. Another difference some have made is a milliner makes custom-designed/commissioned hats where hatters get the final say on the design.

The closest to a gender neutral term in haberdasher. Granted, that term is not limited to hat making. It can be expanded to clothing, accessories, and wears; much like a tailor. Sometimes it can be used to describe someone who supplies cloths and raw materials like that. On occasion, it can be limited to males wares. But based on my personal experience, it is usually gender neutral more often than not.