To simplify my setup using Google TV, I installed a custom launcher and set that as the default one. This removed the horrible home screen and now I just see Jellyfin when the TV turns on. From there, it's a Netflix like experience. Also, the TV remote is able to control the volume of the AVR, so it's a single remote experience. I tried the button remapping route, but abandoned that. The custom launcher is the way to go.
A long time ago I used to use a PC with a remote and Kodi and it was nice, but you'll still need two remotes to get it all working.
Flauncher I didn't care to setup the background so it's a solid color. Soo basic and perfect. https://play.google.com/store/apps/details?id=me.efesser.flauncher