Basically yes. Or: the term is fine; in the eyes of people who never heard or thought about its racist origin.

And your were entirely on the right track when your first comment started with explaining the origin first. When that's your starting point and you then get the response "But I've been saying it for years", it's probably far more successful to go with "Yes, but it can be offensive for people knowing its origin, so why not use available alternative terms?" than with "bro it's been fucking racist for years". In one case people have a high chance on thinking about it, in the other they will instictively feel attacked and get defensive about it.