Technology

You seem to be missing what I'm saying. Maybe a biological comparison would help:

An octopus is extrmely smart, moreso than even most mammels. It can solve basic logic puzzles, learn and navigate complex spaces, and plan and execute different and adaptive stratgies to humt prey. In spite of this, it can't talk or write. No matter what you do, training it, trying to teach it, or even trying to develop an octopus specific language, it will not be able to understand language. This isn't because the octopus isn't smart, its because its evolved for the purpose of hunting food and hiding from predators. Its brain has developed to understand how physics works and how to recognize patterns, but it just doesn't have the ability to understand how to socialize, and nothing can change that short of rewiring its brain. Hand it a letter and it'll try and catch fish with it rather than even considering trying to read it.

AI is almost the reverse of this. An LLM has "evolved" (been trained) to write stuff that sounds good, but has little emphasis on understanding what it writes. The "understanding" is more about patterns in writting rather than underlying logic. This means that if the LLM encounters something that isn't standard language, it will "flail" and start trying to apply what it knows, regardless of how well it applies. In the chess example, this might be, for example, just trying to respond with the most common move, regardless of if it can be played. Ultimately, no matter what you input into it, an LLM is trying to find and replicate patterns in language, not underlying logic.