I understand where you're coming from, but hate is somebody's own fault, not their political team. And many factions not listed as "hate groups" promote violence, such as ANTIFA and certain religious bodies.
Protecting yourself and others from mentally disturbed individuals is necessary, but as soon as we condemn expressions and symbols for no reason other than their being strange to us, we're doing exactly what we want to avoid.
Star Wars explains it best