I bring my yubikey with me, it's in my keychain. This is not only more secure against phone theft/access, which probably is not very relevant for most people, but it spreads the risk of locking yourself out.

For example, I was in Iceland with my girlfriend and she "lost" her phone. We wanted to locate it, so I logged to Google for her, which asked 2FA. If she used her phone, she would have been toast. Instead I made her use yubikeys too, and she just logged in and found her phone.

Obviously you can lose your hardware tokens too, but it's generally less likely (you take out your home keys way less than your phone, for example). You can also backup your TOTP on multiple devices etc., of course.