I think of it exactly in terms of the trolley problem. The whole premise is that if you do nothing (don't vote) more people die. By flipping the lever, fewer people die but you've taken an action that leads directly to their deaths. The philosophical question isn't just "is it better for fewer people to die" but "in pulling the lever, are you directly responsible for those deaths?"
My answer would be that inaction is itself an action. In this scenario, you have found yourself responsible either way. Suppose you pull the lever, though, to save as many lives as you can... Wouldn't the ones who die as a result of this have loved ones that absolutely do blame you?
Contraception is not perfect. If you are on birth control, and you take it every day, you can still get pregnant, it's just unlikely. If you have an IUD, you can still get pregnant, and even worse every pregnancy is more likely to be life threatening. And of course, condoms can break.
As for having less sex ... that's an interesting thing to say ...