I'm surprised how much I'm seeing gyro brought up in these comments. It's a factor, but it's practically negligible. It's all in the steering. Start to tip right, and you'll subconsciously steer slightly to the right to correct your balance. Try to ride as slow as you can and you'll find yourself doing these corrections much more frantically and dramatically. The reason for that is because it takes longer for the wheel to roll under your center gravity and "catch" you when you're going slowly so you have to turn in quicker to maintain balance.

Notice that on almost every bike you see, the front axle on the bike is slightly ahead of the neck's axis of rotation. That offset does two things: 1. It stabilizes the steering so that the bike will tend to steer straight and 2. (more important to my point) It makes the balance-correcting effect of steering more immediate and dramatic, making it much easier to ride at slower speeds.

As a counter argument showing why gyro is barely a factor, these exist:

Edit: if you're not seeing the image like I'm not, Google "ski bike".

Momentum + Gyroscopic effect

how many licks does it take to get a bike to balance its self? The world will never know!

Brah just discovered conservation of angular momentum

I don’t get it when people (usually chavs) can just sit back with their hands in their pockets - when I try it my handlebars twist out to one side instantly.

Fuckin magnets

How do they work?

first, and less importantly, your wheels are gyroscopes

second, and much more importantly, at speed you use your steering to compensate for imbalance. You lean a little right? slight steering to the right compensates. When standing still, steering is no longer an option (duh)

Gyroscope effect. You ever do the experiment where you spin a bike tire really fast and then try to tilt it? Shit's nuts.

It's the central pedal force

Can confirm. Last week, I got home from a ride, stopped in front of the garage, couldn't unclip, and promptly fell over. It turned out one of the bolts fell out from the cleat during the ride, so the cleat just rotated, instead of unclipping. D'oh. Fortunately, I mostly landed in grass, though I did scrape my ankle a bit.

But pedaling on a treadmill make you fall over.

What the hell?

Same principle as a gyroscope: a turning wheel will tend to stay perpendicular or parallel to the direction of the gravity vector because if it starts tilting away from such orientation there's a force that pushes it back.

Also works better with bigger wheels (if I remember it correctly the effect is related to spinning momentum).

I was pretty surprised when learning Physics and they show us how to derive the formula for that (which I totally forgot since that was over 3 decades ago).

Edit: Actually the gyroscopic effetc is just a part of it. See this article

Having the pivot point (steering) for the front wheel behind it's axle helps

A Veritasium video about the topic, if someone is actually interested: https://youtu.be/9cNmUNHSBac