The difference is atheists may be correct, but nowadays whether you're religious isn't, on its own, a huge prediction factor in how you affect others' lives.

When your politics is lib or reactionary, when you're racist, or nationalist, or just deliberately ignorant and loudmouthed about world politics, you are by definition actively and deliberately conspiring to make peoples' lives worse. Those people affected are often minority groups, and/or friends, family and comrades. But whatever kind of human it is, it should be obvious why it's personally important to us to oppose that.