While you're right that that's a downside of downvotes, I think that it's far better than the alternative.

Downvotes means we have a way to discourage really bad behavior and lets others see that it's discouraged. For example, suppose someone posts something bigoted. It sucks to see those kinda comments (especially when they affect you personally). When those comments are heavily downvoted, it feels better, since it tells you that the views expressed in the comment are not acceptable. It's extremely discouraging when I see bigoted posts with a positive score. Without downvoting, they all have positive scores and it's just "less positive".

It'd be nice if reporting was able to remove such comments before anyone sees them, but that will never be the case. Too many communities don't remove comments fast enough and many more simply won't remove comments unless they're really bad, if at all. Some moderators are bigots themselves and others simply don't have the ability to recognize dog whistles that may be in comments. Or they're not personally affected by the malicious comment, so they can be more easily convinced that if the comment was politely worded, it's acceptable even if it's blatantly bigoted.

To be clear, it does suck that users will use it as a disagree button for comments that are otherwise good, but that is far, far worth it. The presence of downvotes were a major reason why I used Reddit (and now this) while disliking the likes of twitter.