148
Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
(garymarcus.substack.com)
this post was submitted on 14 Apr 2024
148 points (94.0% liked)
ChatGPT
8912 readers
1 users here now
Unofficial ChatGPT community to discuss anything ChatGPT
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I think increasingly specialized models and analog systems that run them will be increasingly prevalent.
LLMs at their current scales don't do enough to be worth their enormous cost... And adding more data is increasingly difficult.
That said: the gains on LLMs have always been linear based on recent research. Emergence was always illusory.
I'd like to read the research you alluded to. What research specifically did you have in mind?
Sure: here's the article.
https://arxiv.org/abs/2304.15004
The basics are that:
LLM "emergent behavior" has never been consistent, it has always been specific to some types of testing. Like taking the SAT saw emergent behavior when it got above a certain number of parameters because it went from missing most questions to missing fewer.
They looked at the emergent behavior of the LLM compared to all the other ways it only grew linearly and found a pattern: emergence was only being displayed in nonlinear metrics. If your metric didn't have a smooth t transition between wrong, less wrong, sorta right, and right then the LLM would appear emergent without actually being so.