Except you don't "just get fucked" in those unexpected scenarios because taxis and ubers still exist. Most car owners pay over $1000/month in total cost of ownership, so you'd have plenty of budget for emergency taxis/ubers if you stop owning a car.

The thing is, you're technically correct in your claim that transit time and convenience are the reasons "why people drive". Those are the pros of driving, from the perspective of the individual making the decision to drive rather than use transit. However, in order to understand why we should de-prioritize cars in favor of other forms of transit, we must also consider the cons of driving for the individual (excessive cost, mental fatigue of focusing on driving, elevated risk of bodily harm, etc.) as well as the harms that driving causes to others (pedestrian fatalities, urban sprawl, marginalization of public transport, etc.)

After all, flying a helicopter would be even faster than driving a car, but nobody is going around implying that we should all fly helicopters everywhere just in case we need to rush to our mother's side when she's on her deathbed.

edit: I noticed in another comment you said that taxis don't exist in your city and you avoid using uber out of principle due to their questionable ethics. I would hope that since you are concerned about ethics you would see that car-centric infrastructure has marginalized many people and neighbourhoods, far more than an occasional uber ride ever has. Buying a car and using it daily to avoid the ethical implications of an emergency uber ride once in a while is cutting off your nose to spite your face.