Some rando standing on a street espousing the deranged ramblings of a long-dead dictator is not a legitimate threat in my eyes.

That's exactly the first scene of the Twilight Zone episode He's Alive. Not that it proves anything as it's just a fictional story but it's the first thing I thought of.

An article that also comes to mind is Bartender explains why he swiftly kicks out Nazis even if they're 'not bothering anyone'. Basically, if you allow the "rando" Nazi's a safe place to congregate they'll tell all their friends who tell their friends and eventually your bar, town square, etc are the local Nazi hangout and the extremists start showing up. Now you have too many Nazis to safely and easily remove.

Now, you absolutely don't need to use violence as your language of choice, what's most important is that you make it loud and clear that trying to put down roots is going to be more trouble than it's worth.