As @GreyShuck@feddit.uk correctly highlighted, it wasn't even Latin that displaced so much of the local language. It was the Germanic tribals invading the islands later on. So I'll focus specifically on the Roman role.

The Roman process of Latinisation was rather slow. For reference: Gallia was conquered in 58-50 BCE, but odds are that Gaulish survived until the ~sixth? century of the common era, 600 years later. That's because the Romans didn't really give much of a fuck about what rural local folks spoke - if they rebel you kill them and done, problem solved.

Instead they were actively placing colonies in the conquered regions (to give land to Roman citizens) and converting the local elites to Roman habits and customs, because unlike the farmers the elites could be actually dangerous if rebellious.

The same applies to Britannia. Except that it was conquered ~a century after Gallia, it's a fucking island in the middle of nowhere with harder access, it doesn't grow grapes or olives, grain production in the Empire was mostly in Africa and Egypt so odds are that they couldn't reliably grow their wheat variety there either... really, the island was mostly a tin mining outpost.

Another factor to consider is the distribution of the Roman settlements in the area:

Are you noticing a pattern? Most settlements were in the Southeast, specially the larger ones (in yellow, full of Roman citizens). Perhaps not surprisingly the extant Brittonic languages are spoken further West, when you couple this with the tribal invasions. (That's simply because of the Fretum Oceani aka Strait of Dover. It was easier to reach the island by there.)

Here is the crazy thing about empires vs nation states. In general because empires hold vast swaths of territory. It is very common for them to be somewhat tolerant of the various groups under their control. The goal is wealth extraction not conformity. So if you let the locals keep most of their culture then they are generally more productive and less likely to get uppity.

Nation states are smaller and tend to lean on a cohesive identity to maintain power. So it is much more important for everyone to be the same. Otherwise it could lead to balkanization.

I believe that the language was transmitted with trade rather than conquest in the roman empire. At worse one would have a translator to receive orders and communicate with Rome. But with becoming part of the empire, trade develop, and all the people engaged in the trading will start to learn the language.

For that England was not well placed. It was described as a poor land by the Romans, barely worth a conquest. So I expect Rome didn't care enough to develop it.

It should be noted that usually when we talk about languages of the past, we're talking about the languages of the aristocracy. In rural areas, it vary wildly depending on the history of the place and the ties they had with the nearest city. No one cared about the language they talked as long as one guy there could translate for the others. They didn't write book either. It's only recently in history, with the idea of nation and the school for everyone, that people started to talk the same language on all of a country. You may have heard about the literacy of a population that was very low until the 60's or 70's. This is actually when people all started to actually talk the same language, even in remote places.

I understand that the Romans were unable to conquer Scotland so they build Hadrian’s Wall (which explains the survival of older cultures there).

Keep in mind it was less "unable to" and more "not interested enough to". They could have absolutely taken Scotland if they poured enough legions into it, but then what, oats? Hadrian's wall and all the other parallel walls were basically a way to keep the neighbors in check so they could send more troops to places like the Persian border they cared about a lot.

The Romans had an impact to a greater or lesser degree across the whole of the area that they controlled in Great Britain, including Cornwall and Wales, but the Brythonic (Celtic) culture seems to survive for most ordinary people throughout that time. It was really only the arrival of the Germanic peoples - the Angles and Saxons - that seemed to displace the Brythonic language and culture from much of the lands that they went on to occupy, which was largely the land that was easier to work in the majority of England, but not the more difficult land in the West and North - including Cornwall and Wales.

Around that time, there is evidence that some Brythonic speakers were moving into Wales - presumably from England - causing changes to the existing dialects there, also some Britons seem to have migrated to Brittany on the continent, and there was an outbreak of plague that affected much of the Roman lands and caused a population decline there - but less so among the Germanic people.

No matter which had more effect, it was the Germanic people and culture that displaced the existing one - not the Romans.