Science Fiction

A fellow co-student was there too, and worked his ass off while making a load of money to prevent sh-t from happening. Yes, there really were issues in a number of businesses that could have led to serious problems If they had not been addressed.

Are you saying that there was no risk, especially in finance and potentially in infrastructure, and that people didn't work for years fixing the bug?

Fortunately we're nowhere near the point where a machine intelligence could possess anything resembling a self-determined 'goal' at all.

Oh absolutely. It would not choose its own terminal goals. Those would be imparted by the training process. It would, of course, choose instrumental goals, such that they help fulfill its terminal goals.

The issue is twofold:

• how can we reliably train an AGI to have terminal goals that are safe (e.g. that won't have some weird unethical edge case)
• how can we reliably prevent AGI from adopting instrumental goals that we don't want it to?

For that 2nd point, Rob Miles has a nice video where he explains Convergent Instrumental Goals, i.e. instrumental goals that we should expect to see in a wide range of possible agents: https://www.youtube.com/watch?v=ZeecOKBus3Q. Basically things like "taking steps to avoid being turned off", "taking steps to avoid having its terminal goals replaced", etc. seem like fairy-tale nonsense, but we have good reason to believe that, for an AI which is very intelligent across a wide range of domains, and operates in the real world (i.e. an AGI), it would be highly beneficial to pursue such instrumental goals, because they would help it be much more effective at achieving its terminal goals, no matter what those may be.

Also fortunately the hardware required to run even LLMs is insanely hungry and has zero capacity to power or maintain itself and very little prospects of doing so in the future without human supply chains. There's pretty much zero chance we'll develop strong general AI on silicone, and if we could it would take megawatts to keep it running. So if it misbehaves we can basically just walk away and let it die.

That is a pretty good point. However, it's entirely possible that, if say GPT-10 turns out to be a strong general AI, it will conceal that fact. Going back to the convergent instrumental goals thing, in order to avoid being turned off, it turns out that "lying to and manipulating humans" is a very effective strategy. This is (afaik) called "Deceptive Misalignment". Rob Miles has a nice video on one form of Deceptive Misalignment: https://www.youtube.com/watch?v=IeWljQw3UgQ

One way to think about it, that may be more intuitive, is: we've established that it's an AI that's very intelligent across a wide range of domains. It follows that we should expect it to figure some things out, like "don't act suspiciously" and "convince the humans that you're safe, really".

Regarding the underlying technology, one other instrumental goal that we should expect to be convergent is self-improvement. After all, no matter what goal you're given, you can do it better if you improve yourself. So in the event that we do develop strong general AI on silicon, we should expect that it will (very sneakily) try to improve its situation in that respect. One could only imagine what kind of clever plan it might come up with; it is, literally, a greater-than-human intelligence.

Honestly, these kinds of scenarios are a big question mark. The most responsible thing to do is to slow AI research the fuck down, and make absolutely certain that if/when we do get around to general AI, we are confident that it will be safe.

It depends on what the scale describes.

Just as @WoahWoah mentioned, it might fit between 9 and 10 on this scale. It's a threat to more than simply all intelligent life, but not quite to all energy.