37
submitted 2 months ago* (last edited 2 months ago) by Hackworth@lemmy.world to c/technology@lemmy.world

“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”

The paper, "Emergent Representations of Program Semantics in Language Models Trained on Programs" can be found here.

Abstract

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.

you are viewing a single comment's thread
view the rest of the comments
[-] atrielienz@lemmy.world 13 points 2 months ago

I'm actually pretty sure the downvotes are because LLM's don't think. They don't even process. They pick the highest number and spit out the information attached to it.

[-] Deceptichum@quokk.au 1 points 2 months ago

Do I “think” or does my brain pick the closest neuron and spit out a function based on that input?

If we could recreate the universe, would I do the exact same thing in the exact same situation?

[-] atrielienz@lemmy.world 10 points 2 months ago

I'm sorry. Because you don't understand how your brain works you're suggesting that it must work in the same way as something a similar brain created because you don't know how either thing works. That's not an argument.

[-] Deceptichum@quokk.au -4 points 2 months ago* (last edited 2 months ago)

No, I’m not suggesting that.

I’m suggesting that if we don’t even understand how consciousness works for ourselves, we cannot make claims about how it will look for other things.

Deterministically free will does not exist, if we cannot exercise free will we cannot have independent thoughts just the same as a machine.

Truth is we don’t really know shit, we’re biological machines that are able to think they’re in control of themselves based on inputs. If we ever discover true AGI it will be on accident as we fiddle with technologies such as LLMs or any other complex models.

[-] atrielienz@lemmy.world 5 points 2 months ago

Okay. Feed a new species that hasn't been named yet into an LLM. Does it name that new creature? Can it decide what family or phylum etc it belongs? Does it pick up the specific attributes of that new species?

[-] Deceptichum@quokk.au 0 points 2 months ago* (last edited 2 months ago)

It might be able to pick those things out, I certainly couldn't.

Edit: So ChatGPT correctly identified a new species from 4 days ago as a type of Storm Petrel and a new flower from Sri Lanka as an Orchidaceae. Far better than I could do.

[-] atrielienz@lemmy.world 4 points 2 months ago

That is very deliberately not in the spirit of the question I asked. It's almost like you're intent on misunderstanding on purpose just so you can feel like you're right.

[-] Deceptichum@quokk.au 1 points 2 months ago

You asked if it could do I task I wasn’t even capable of doing, and this was your assessment of consciousness.

[-] atrielienz@lemmy.world 3 points 2 months ago* (last edited 2 months ago)

No. I asked if it had been given an unclassified un-named species. Not something someone else just discovered and has already parsed information on. And the point is humans can and do do this, have done it for centuries with the right training as those systems we use for classification have been dialed in.

The model has the information on how to classify. It can be added to with scraped data from the internet. But it does not do the same things a trained individual does to classify and name a new species. Because it is not capable of that.

[-] Deceptichum@quokk.au 1 points 2 months ago* (last edited 2 months ago)

The information from 4 days was not parsed on, that’s why I chose something so recent.

And LLM can be trained to do this. Literally when it looked at the Petrel it did things humans do such as take note of the dark colours common in seabirds, the small size, etc. and it used those points to reach its conclusion.

We don’t do anything special as humans, we take in data, process it, and spit out a result. It’s why a child has to be taught basic concepts such as creativity or socialising.

[-] atrielienz@lemmy.world 2 points 2 months ago* (last edited 2 months ago)

Given nothing at all, could the LLM quantify or develop the tools and systems we use to categorize such species? Could it discover a species? The spirit of the question is, humans have been able to look at the world around them, using data we gain from our 5 senses and the scientific method to do this. The LLM cannot develop the same information gathering or classification, diagnostic, or scientific method skills in order to do the same. It relies solely on what we provide it and can only operate within those parameters. It does not have senses of its own. That's the point. Go look up how we have learned to quantify sapience. Because what you're saying is that you (a small data point out of trillions or more) can't do a thing a computer can do, so it must be able to think.

[-] sugar_in_your_tea@sh.itjust.works 2 points 2 months ago

Exactly. If we give an LLM with no training data a large group of specimens, will it organize them into logical groups? Does it even understand the concept of organizing things into discrete groups?

That's something that's largely encoded into our brain structures due to millennia of evolution (or creation, take your pick) where such organization is advantageous. The LLM would only do it if we indicated that such organization is advantageous, and even then would only do it if we gave it a desired output. An LLM will only reflect the priorities of its creator, or at least the priorities baked in to the training data. It's not going to suggest that something else entirely be considered, because it only considers things from the lenses we give it.

Humans will question assumptions, will organize things without being prompted, and will generate our own priorities. I firmly believe an LLM cannot, and thus cannot be considered self-deterministic, and thus not sentient. All it can do is optimize for the priorities we give it, and while it may do that in surprising ways, that doesn't mean there's "thinking" going on, just that it's a complex system we don't fully understand (even if we created it). Maybe human brains work in a similar way (i.e. completely deterministic given a specific genome and "training data"), but we know LLMs work that way, so until we prove that humans work similarly, we cannot equate them. It's kind of like the P = NP question, we know LLMs are deterministic, we don't know if humans are. So the question isn't "can LLMs think" (we know they can't), but "can humans think."

[-] technocrit@lemmy.dbzer0.com 3 points 2 months ago* (last edited 2 months ago)

I’m suggesting that if we don’t even understand how consciousness works for ourselves, we cannot make claims about how it will look for other things.

So is a rock conscious? I guess we'll never know... But AI!?! Definitely conscious! smh.

[-] tabular@lemmy.world 3 points 2 months ago* (last edited 2 months ago)

I suspect others are talking about "thinking" only objectively.

B) If a LLM had a subjective experience when given input presumably it has none when all processes are stopped (subjectively, unverifiable).

A) If a LLM has no input then there are no processes going on at all which could be described as thinking (objectively, verifiable: what is the program doing).

[-] A_A@lemmy.world -5 points 2 months ago

Science cannot say much about what it is to think since it doesn't understand the brain well enough ... and the day we can fully explained it, we will also be able to replicated it on computers.

[-] atrielienz@lemmy.world 4 points 2 months ago

Science can and does quantify what our brains do vs what an LLM does though. That's the point. That's why the brain knows when it's supplying wrong information or guessing but the LLM does not.

The LLM can provide wrong information. What it can't do is intentionally lie.

[-] A_A@lemmy.world -3 points 2 months ago

i agree with you that we are much better than LLMs in the fact we can verify our errors (and we can do much more things LLMs don't do).
Still i am happy to have access to their vast memory and i know where they fail most of times so i can work with them in a productive way.
The day we provide them (or DNNs) with "Will" is i think when they will become (more) dangerous.

[-] technocrit@lemmy.dbzer0.com 1 points 2 months ago* (last edited 2 months ago)

Gotta use extra heavy quotes around "will"... Mixing up AI pseudo-science with fascist pseudo-science.

[-] atrielienz@lemmy.world 0 points 2 months ago* (last edited 2 months ago)

Nah fam. You're comparing this thing to yourself and other humans because to you it sounds like us. "Sounds like" is the phrasing conspiracy theorists use to try to make correlations regardless of whether they make sense or not without filling in any actual facts. Either you know and understand how they work and are researching this and have a valid rebuttal or you don't. But an assumption does not make anything you said fact. People who are researching this have already refuted this claim.

You don't even know what the word "will" would mean in this context, or that it would provide an LLM with the kind of consciousness required to be sapient. So can we stop arguing if you just admit that you like using LLM's so you have a bias and a poor understanding of what they are and aren't.

[-] technocrit@lemmy.dbzer0.com 2 points 2 months ago* (last edited 2 months ago)

Science cannot say much about what it is to think since it doesn’t understand the brain well enough

Wild pseudo-scientific generalization.

and the day we can fully explained it, we will also be able to replicated it on computers.

There are many many things that are fully explained but will never be replicated on computers. Eg. Any numerical problem bigger than a computer.

this post was submitted on 24 Aug 2024
37 points (63.9% liked)

Technology

59161 readers
2253 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS