Right-click on the Sound icon, then Sound Settings.

You get to: System > Sound

You can do stuff directly in the first panel, but scroll down:

System > Sound > All Sound Devices

Find your Display there, it could look like:

Click on it.

The first option in the detail view is to toggle it as a sound "output" device:

The options are "Allow" or "Don't allow". Once it's blocked, it's no longer in the list of options for switching easily between sound output devices (automatically or manually).