I believe this can happen when YouTube doesn't detect hardware playback capabilities right. It gets stuck in mp4 mode which is often limited to 720p.
My Steam Deck will play 4k60 video just fine so this seems to be a problem specifically on your end. YouTube will let you play videos of higher resolution than your screen offers regardless of you device (something I use often to get decent 1080p video out of the compressed trash YouTube pushes, by selecting 1440p or 4k on my 1080 devices).
Possible fixes I can think of:
- Try disabling all addons (yes, including adblock) to see if an addon is causing this. Works fine for me with uBlock, but who knows
- Try resetting your Firefox profile. Sometimes a weird setting can break browsers in spectacular ways.
- You're probably using Flatpak for Firefox. You may need to enable hardware acceleration in a tool like Flatseal for high performance video playback.