I’ve had similar issues in the past.

You can restart the HomePods and your AppleTV should become the hub.

It should stay the hub until you restart or have a power outage.

IMO the hub should prioritize anything with a wired internet connection Apple TVs then HomePods last.