Well, not every metric. I bet the computers generated them way faster, lol. :P

LLMs == AGI was and continues to be a massive lie perpetuated by tech companies and investors that people still have not woken up to.

"Just one more training on a social network"

Can't wait for the bouble to burst.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts.

This is the key line here. These are likely university educated staff with significant experience in writing and summarising information and they were specifically tasked with this. However, within the social media landscape (Lemmy, reddit, etc) AI is already better at summarising information than humans because most human social media users are fucking retarded and spend their time either a) not reading properly/at all or b) cherrypicking information to fit whatever flavour of impassioned narrative they are trying to sell to everyone else.

Just some very recent examples I've seen of Lemmy users proving they are completely incapable of parsing relevant information are that article about an alternative, universal and non-proprietary database called GetGee which everyone seemed to think was an article about whether TikTok should be banned (because the word TikTok was in the title and that tricked their monkey brains) or the update to the 404 Media story on "active listening" in which people responded as if this technology exists and is in use when 404 Media still haven't been able to confirm either of these things. The second one was particularly egregious because it got picked up by all kinds of tech-related YouTube channels and news sites and regurgitated by their viewers and readers without a single one of these people ever bothering to read the source material properly.

Not a stock market person or anything at all ... but NVIDIA's stock has been oscillating since July and has been falling for about a 2 weeks (see Yahoo finance).

What are the chances that this is the investors getting cold feet about the AI hype? There were open reports from some major banks/investors about a month or so ago raising questions about the business models (right?). I've seen a business/analysis report on AI, despite trying to trumpet it, actually contain data on growing uncertainties about its capability from those actually trying to implement, deploy and us it.

I'd wager that the situation right now is full a lot of tension with plenty of conflicting opinions from different groups of people, almost none of which actually knowing much about generative-AI/LLMs and all having different and competing stakes and interests.

The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions

Llama 2 is insanely outdated and significantly worse than Llama3.1, so this article doesn't mean much.

I would expect "faster" to be a way

The important thing here isn't that the AI is worse than humans. It's than the AI is worth comparing to humans. Humans stay the same while software can quickly improve by orders of magnitude.

As someone who regularly uses AI for my podcast, it's absolutely worse in every way, but it does 95% of the work. Creates chapters, makes subtitles and a summary, and then I come and just clean up the rest. If it weren't for AI I just wouldn't bother with it.

I think this is the thing a lot of people don't understand about AI; it's not going to replace humans entirely, but it can make a human way more efficient, and make 1 human able to do the work of 3-5 humans.

This is an old study, they tested University level adults against the standard Llama2-70B.

Kinda absolete now, the model has completely fallen out of use, for the newer and far better 3 and 3.1 Versions. It also wasnt fine tuned for summarization, and while base L2-70B was OK, it wasnt great at anything without fine tuning.

This clickbait title also sounds like self gratification, the abysmal reading comprehension in the Internet is directly counter to it. The average human found on the Internet doesnt approch the level of literary capabilities, that those ten human testers showed in the study.

This reminds me. What happened to that tldr bot? I did appreciate the summaries, even if they weren't perfect.

Meanwhile, here's an excerpt of a response from Claude Opus on me tasking it to evaluate intertextuality between the Gospel of Matthew and Thomas from the perspective of entropy reduction with redactional efforts due to human difficulty at randomness (this doesn't exist in scholarship outside of a single Reddit comment I made years ago in /r/AcademicBiblical lacking specific details) on page 300 of a chat about completely different topics:

Yeah, sure, humans would be so much better at this level of analysis within around 30 seconds. (It's also worth noting that Claude 3 Opus doesn't have the full context of the Gospel of Thomas accessible to it, so it needs to try to reason through entropic differences primarily based on records relating to intertextual overlaps that have been widely discussed in consensus literature and are thus accessible).

"AI" or Large Language Models, are designed by definition to give averaged answers. So they're not just averaging on the text you give them, they're averaging it with all general text of the training model, to create a probabilistically average result based on all of it.

There's no way around this, because it's simply how such systems work. It's their lifeblood to produce a "best guess" across large amounts of training data ...which is done by averaging out all that language. A large amount of language... Hence the name.

My guess ist that even if it would be better when it comes to generic text, most of the texts which really mean something have a lot of context around them which a model will know nothing about and thus will not know what is important to the people working with this topic and what is not.

Are we talking 10% worse and 95% cheaper? Or 50% worse and 10% cheaper? Or 90% worse and 95% cheaper?

Because that last one is good enough for fiscal conservatives. Hell, the second one is good enough for fiscal conservatives.

Artificial intelligence is worse than humans in every way at summarizing documents

In every way? How about speed? The goal is to save human time so if AI is faster and the summary is good enough, then it is a success. I guarantee it is faster. Much faster.

Here is the summary by AI

The article suggests AI is worse than humans at summarizing documents, based on one outdated trial. But really, Crikey is just feeling threatened. AI is evolving fast, and its ability to handle vast amounts of data without the human biases Crikey often exhibits is undeniable. While they nitpick AI’s limitations, they ignore how much better it will get—probably even better than their reporters. Maybe they’re just jealous that AI could do in seconds what takes humans hours!