289
'LLM-free' is the new '100% organic' - Creators Are Fighting AI Anxiety With an ‘LLM-Free’ Movement
(www.theatlantic.com)
this post was submitted on 13 Jun 2024
289 points (100.0% liked)
Technology
37707 readers
394 users here now
A nice place to discuss rumors, happenings, innovations, and challenges in the technology sphere. We also welcome discussions on the intersections of technology and society. If it’s technological news or discussion of technology, it probably belongs here.
Remember the overriding ethos on Beehaw: Be(e) Nice. Each user you encounter here is a person, and should be treated with kindness (even if they’re wrong, or use a Linux distro you don’t like). Personal attacks will not be tolerated.
Subcommunities on Beehaw:
This community's icon was made by Aaron Schneider, under the CC-BY-NC-SA 4.0 license.
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Except when it comes to LLM, the fact that the technology fundamentally operates by probabilisticly stringing together the next most likely word to appear in the sentence based on the frequency said words appeared in the training data is a fundamental limitation of the technology.
So long as a model has no regard for the actual you know, meaning of the word, it definitionally cannot create a truly meaningful sentence. Instead, in order to get a coherent output the system must be fed training data that closely mirrors the context, this is why groups like OpenAi have been met with so much success by simplifying the algorithm, but progressively scrapping more and more of the internet into said systems.
I would argue that a similar inherent technological limitation also applies to image generation, and until a generative model can both model a four dimensional space and conceptually understand everything it has created in that space a generated image can only be as meaningful as the parts of the work the tens of thousands of people who do those things effortlessly it has regurgitated.
This is not required to create images that can pass as human made, but it is required to create ones that are truely meaningful on their own merits and not just the merits of the material it was created from, and nothing I have seen said by experts in the field indicates that we have found even a theoretical pathway to get there from here, much less that we are inevitably progressing on that path.
Mathematical models will almost certainly get closer to mimicking the desired parts of the data they were trained on with further instruction, but it is important to understand that is not a pathway to any actual conceptual understanding of the subject.
What you're describing is Markov chain, not an LLM.
It does, that's like the entire point of word embeddings.
Generally the term Markov chain is used to discribe a model with a few dozen weights, while the large in large language model refers to having millions or billions of weights, but the fundamental principle of operation is exactly the same, they just differ in scale.
Word Embeddings are when you associate a mathematical vector to the word as a way of grouping similar words are weighted together, I don’t think that anyone would argue that the general public can even solve a mathematical matrix, much less that they can only comprehend a stool based on going down a row in a matrix to get the mathematical similarity between a stool, a chair, a bench, a floor, and a cat.
Subtracting vectors from each other can give you a lot of things, but not the actual meaning of the concept represented by a word.
LLMs rely on billions of precise calculations and yet they perform poorly when tasked with calculating numbers. Just because we don't calculate anything consciously to get a meaning of a word doesn't mean that no calculations are actually done as part of our thinking process.
What's your definition of "the actual meaning of the concept represented by a word"? How would you differentiate a system that truly understands the meaning of a word vs a system that merely mimics this understanding?
No part of a human or animal brain operates on subtracting tables of cleanly defined numbers from each other so I think it’s pretty safe to say that no matrix calculation is done on a handful of numbers as part of much less as our sole means of understanding concepts or objects.
I don’t know exactly how one could tell true understanding from minicry, far smarter and more well researched people than me have debated that for decades, i’m just pretty sure what we think an kindness is boils down to something a bit more complex than a high school math problem discribing a word cloud.
So you're basically saying that, in your opinion, tensor operations are too simple of a building block for understanding to ever appear out of them as an emergent behavior? Do you feel that way about every mathematical and logical operation that a high school student can perform? That they can't ever in whatever combination create a system complex enough for understanding to emerge?
They are definitely to simple to represent the entirety of an concepts meaning on their own. Yep, I don’t believe it’s likely that such an incrediblely intricate thing as a nuron, much less the idea of conceptual meaning, can be replicated by a high school math problem. Maybe they could be a part, but your off by about a half a dozen order of magnitude at least from where we are now with love being a matrix with a few hundred numbers in it.