Except LLM output is largely gibberish. Just confident gibberish. There’s a reason we call it “AI slop”.
LLM responses are only ever “sound” when they’re regurgitating existing information they were trained on. Beyond some simple transformations, they are unable to create original ideas. They very frequently break down on somewhat unique tasks, as evidenced by the ever-prevalent code-slop which is eroding our software.
They don’t have a memory of previous conversations (unless you literally copy-paste it into the prompt), they don’t learn (Claude “memories” is literally just copy-pasting a summary into the prompt, only automatically). They don’t have any “thoughts” of their own between prompts (OpenClaw just keeps prompting them to pretend they are autonomous).
The underlying implementation of “reasoning” in LLMs is literally “hallucinate some more text which vaguely looks like thoughts and hope that influences the answer”. LLMs are probabilistic models which we figured out how to make so they produce somewhat correct-looking answers at a rate a little higher than chance.
Magic 8-balls sometimes give sound responses. Do they think? Where do we draw the line with this interpretation of “thinking”?
I would disagree with you, and would suspect you are basing your assessment of their abilities on dated usage. I hold a MSc from what is arguably the most prestigious University in Europe, in regards to computer science, and my major was in AI. Believe me when I say I know exactly how they function.
I still assert you are oversimplifying their current capabilities, and seem to be conflating LLM with Markov Chains. LLM do not simply regurgitate existing content, and are in fact capable of creating wholly new content not seen before. Hallucinations occur when their context buffer is too small, and as time goes on, it will largely be a thing of the past.
Magic Eight Balls, as I’m sure you’re aware, have a limited, predetermined number of responses. They are in no way comparable. LLM use the equivalent of synapses, just digital whereas we use biological, but the function is the same. Modern AI is distinguishable only by the medium used, silicon versus organic material. As the number of input parameters, and context windows grows, the difference between them and our own brains will shrink until the medium is the only remaining difference.
We’re not there yet, but I would argue they are already capable of thought if we define that to mean reasoning towards a response using all available information, instead of taking a predetermined or random path to one. We draw the line at biological life and LLM, nothing else we are aware of can think.
I hold a MSc from what is arguably the most prestigious University in Europe
Good for you. Have a cookie, I guess?
LLM do not simply regurgitate existing content, and are in fact capable of creating wholly new content not seen before.
Citation needed.
Hallucinations occur when their context buffer is too small, and as time goes on, it will largely be a thing of the past.
A whole book of citations needed. That claim is wildly inconsistent with the consensus about AI hallucinations.
Magic Eight Balls, as I’m sure you’re aware, have a limited, predetermined number of responses.
You mean like how LLMs keep hallucinating the same passwords and nonexistent dependencies to the point that bad actors are using that fact to compromise vibe coded systems via techniques like slopsquatting?
I would disagree with you, and would suspect you are basing your assessment of their abilities on dated usage.
In fact, I keep experimenting with frontier models (including Fable when it was available) just so that the “but we’ve made so much progress in the past few months” argument can’t be used against me. You’re wildly overselling their capabilities.
Thanks, I like cookies. You should have one too for participating in this discourse, Internet stranger. Help yourself!
I’m citing myself. If you have information to the contrary of something I’ve said, feel free to provide it.
Those examples are no different than instincts in people. The “training data” will shine through, but it doesn’t preclude new behavior.
… arguments can’t be used against me.
This reads to me like you went into the interaction with a prescribed expectation, and were using it only to validate your prejudice. If you used Fable, and didn’t think any thought was occurring, you’re either being purposefully obtuse or we have wildly different definitions of thinking.
Why don’t you give me your definition that includes people but excludes advanced LLM such as Fable?
I think the Wikipedia definition of thought is quite good.
However, I have a feeling whatever definition I came up with, you’d just claim LLMs fit into it because their output is sometimes somewhat coherent.
You can claim that technically LLMs “think” because the output text sometimes contains conclusions, and sometimes they’re even rational, even though the LLMs still struggle with counting Rs in “strawberry”.
I find that disingenuous because it implies that the LLM is in any way aware of anything, that it can passively form ideas.
Most importantly, it implies that you can trust it for even basic reasoning. That you can trust the plagiarism machine that tells you that you should put glue on your pizza, eat rocks and walk to the car wash instead of driving, or that you will be able to trust it at some point in the future.
Whatever definition of thinking we use, it should include a simple rule - that the allegedly thinking entity should demonstrate that intelligence by being able to reliably answer simple queries correctly. Humans, by and large, can do that. LLMs fail at it miserably. If the LLMs were truly thinking, that should be shocking. Understanding the underlying technology - and that it is not truly reasoning - makes it obvious and expected.
Even OpenAI admitted hallucinations are an unfixable mathematical inevitability - something you handwaved as a matter of time to fix. No, the fact that humans can have hallucinations is not comparable.
Cognitive process in which the mind considers, creates, or manipulates ideas, representations, or information.
Yes, you are correct - that is exactly what LLM do.
You make it sound as if 99% of the time the output of an LLM is gibberish, and only 1% it is coherent, but in my personal experience it is the exact opposite ratio. However, I can’t account for your experiences, as I am very aware that both expectation and perception are the basis for reality, and yours are obviously tuned negatively. You see what you expect to see, and if others have drastically different experiences that is something you cannot fathom as you have only your own to draw from anecdotally.
An LLM certainly could passively form ideas, if it was allowed to passively execute and output it’s thoughts to a console. That is not typically how they are used, though, and have been trained to output one response for one input. The capability is there, though.
I trust AI far more than I do a random person. They have access to far more information, and are more likely to be correct about any particular question asked. If you need a specialist, a human will usually be superior to the AI if you seek one out, but the next best thing is an AI - not a random person.
If you’re going to claim that hallucinations are an inevitably, please cite your source. My sincere understanding is that hallucinations occur when an LLM lacks context, and rather than seeking it, it takes a shortcut by using a random value for the missing information. This is due to a combination of a limited context window, because of memory constraints, and performance considerations as the operators want faster responses over thorough, comprehensive thinking as it aligns with cost savings concerns from the business perspective.
The advent of MoE (Mixture of Experts) has cut down on hallucinations considerably, by reducing the footprint of the thinking model to only that which is relevant to the prompt, which frees up memory that can be allocated to context. Similar incremental improvements will inevitably make hallucinations a funny story from the past, which I believe is the inevitability.
If you have some mathematical formula that proves hallucinations are unsolvable, I’d be happy to read it and reflect upon it. However, I strongly suspect that any formula you produce is only relevant to the technology of LLMs as they operate today, and that such problems are indeed solvable, and in no way inevitable.
Your derisive pet names for the technology make it clear you’ve already made up your mind, though, so don’t feel obligated to continue to argue the case that no one will ever solve what you believe is unsolvable.
I trust AI far more than I do a random person. They have access to far more information, and are more likely to be correct about any particular question asked.
That is a terrifying stance. And, frankly, embarrassing.
OpenAI, the creator of ChatGPT, acknowledged in its own research that large language models will always produce hallucinations due to fundamental mathematical constraints that cannot be solved through better engineering, marking a significant admission from one of the AI industry’s leading companies.
[…] The research proposed “explicit confidence targets” as a solution, but acknowledged that fundamental mathematical constraints meant complete elimination of hallucinations remained impossible.
I, of course, disagree. Humans on the whole are ignorant and the average person is insufferably ill-informed. You are proving to be the exception. In general however, people suck at answering questions on subjects in which they’re not experts.
I have read that study, but I believe it to be flawed and I’m not alone. Please read the counter argument that hallucination is a structural phenomenon of estimation itself.
This reframes hallucination as structural misalignment between loss minimization and human-acceptable outputs, and hence estimation errors induced by miscalibration.
In other words, it is a problem that can be solved.
Except LLM output is largely gibberish. Just confident gibberish. There’s a reason we call it “AI slop”.
LLM responses are only ever “sound” when they’re regurgitating existing information they were trained on. Beyond some simple transformations, they are unable to create original ideas. They very frequently break down on somewhat unique tasks, as evidenced by the ever-prevalent code-slop which is eroding our software.
They don’t have a memory of previous conversations (unless you literally copy-paste it into the prompt), they don’t learn (Claude “memories” is literally just copy-pasting a summary into the prompt, only automatically). They don’t have any “thoughts” of their own between prompts (OpenClaw just keeps prompting them to pretend they are autonomous).
The underlying implementation of “reasoning” in LLMs is literally “hallucinate some more text which vaguely looks like thoughts and hope that influences the answer”. LLMs are probabilistic models which we figured out how to make so they produce somewhat correct-looking answers at a rate a little higher than chance.
Magic 8-balls sometimes give sound responses. Do they think? Where do we draw the line with this interpretation of “thinking”?
I would disagree with you, and would suspect you are basing your assessment of their abilities on dated usage. I hold a MSc from what is arguably the most prestigious University in Europe, in regards to computer science, and my major was in AI. Believe me when I say I know exactly how they function.
I still assert you are oversimplifying their current capabilities, and seem to be conflating LLM with Markov Chains. LLM do not simply regurgitate existing content, and are in fact capable of creating wholly new content not seen before. Hallucinations occur when their context buffer is too small, and as time goes on, it will largely be a thing of the past.
Magic Eight Balls, as I’m sure you’re aware, have a limited, predetermined number of responses. They are in no way comparable. LLM use the equivalent of synapses, just digital whereas we use biological, but the function is the same. Modern AI is distinguishable only by the medium used, silicon versus organic material. As the number of input parameters, and context windows grows, the difference between them and our own brains will shrink until the medium is the only remaining difference.
We’re not there yet, but I would argue they are already capable of thought if we define that to mean reasoning towards a response using all available information, instead of taking a predetermined or random path to one. We draw the line at biological life and LLM, nothing else we are aware of can think.
Good for you. Have a cookie, I guess?
Citation needed.
A whole book of citations needed. That claim is wildly inconsistent with the consensus about AI hallucinations.
You mean like how LLMs keep hallucinating the same passwords and nonexistent dependencies to the point that bad actors are using that fact to compromise vibe coded systems via techniques like slopsquatting?
In fact, I keep experimenting with frontier models (including Fable when it was available) just so that the “but we’ve made so much progress in the past few months” argument can’t be used against me. You’re wildly overselling their capabilities.
Thanks, I like cookies. You should have one too for participating in this discourse, Internet stranger. Help yourself!
I’m citing myself. If you have information to the contrary of something I’ve said, feel free to provide it.
Those examples are no different than instincts in people. The “training data” will shine through, but it doesn’t preclude new behavior.
This reads to me like you went into the interaction with a prescribed expectation, and were using it only to validate your prejudice. If you used Fable, and didn’t think any thought was occurring, you’re either being purposefully obtuse or we have wildly different definitions of thinking.
Why don’t you give me your definition that includes people but excludes advanced LLM such as Fable?
I think the Wikipedia definition of thought is quite good.
However, I have a feeling whatever definition I came up with, you’d just claim LLMs fit into it because their output is sometimes somewhat coherent.
You can claim that technically LLMs “think” because the output text sometimes contains conclusions, and sometimes they’re even rational, even though the LLMs still struggle with counting Rs in “strawberry”.
I find that disingenuous because it implies that the LLM is in any way aware of anything, that it can passively form ideas.
Most importantly, it implies that you can trust it for even basic reasoning. That you can trust the plagiarism machine that tells you that you should put glue on your pizza, eat rocks and walk to the car wash instead of driving, or that you will be able to trust it at some point in the future.
Whatever definition of thinking we use, it should include a simple rule - that the allegedly thinking entity should demonstrate that intelligence by being able to reliably answer simple queries correctly. Humans, by and large, can do that. LLMs fail at it miserably. If the LLMs were truly thinking, that should be shocking. Understanding the underlying technology - and that it is not truly reasoning - makes it obvious and expected.
Even OpenAI admitted hallucinations are an unfixable mathematical inevitability - something you handwaved as a matter of time to fix. No, the fact that humans can have hallucinations is not comparable.
Wikipedias definition:
Yes, you are correct - that is exactly what LLM do.
You make it sound as if 99% of the time the output of an LLM is gibberish, and only 1% it is coherent, but in my personal experience it is the exact opposite ratio. However, I can’t account for your experiences, as I am very aware that both expectation and perception are the basis for reality, and yours are obviously tuned negatively. You see what you expect to see, and if others have drastically different experiences that is something you cannot fathom as you have only your own to draw from anecdotally.
An LLM certainly could passively form ideas, if it was allowed to passively execute and output it’s thoughts to a console. That is not typically how they are used, though, and have been trained to output one response for one input. The capability is there, though.
I trust AI far more than I do a random person. They have access to far more information, and are more likely to be correct about any particular question asked. If you need a specialist, a human will usually be superior to the AI if you seek one out, but the next best thing is an AI - not a random person.
If you’re going to claim that hallucinations are an inevitably, please cite your source. My sincere understanding is that hallucinations occur when an LLM lacks context, and rather than seeking it, it takes a shortcut by using a random value for the missing information. This is due to a combination of a limited context window, because of memory constraints, and performance considerations as the operators want faster responses over thorough, comprehensive thinking as it aligns with cost savings concerns from the business perspective.
The advent of MoE (Mixture of Experts) has cut down on hallucinations considerably, by reducing the footprint of the thinking model to only that which is relevant to the prompt, which frees up memory that can be allocated to context. Similar incremental improvements will inevitably make hallucinations a funny story from the past, which I believe is the inevitability.
If you have some mathematical formula that proves hallucinations are unsolvable, I’d be happy to read it and reflect upon it. However, I strongly suspect that any formula you produce is only relevant to the technology of LLMs as they operate today, and that such problems are indeed solvable, and in no way inevitable.
Your derisive pet names for the technology make it clear you’ve already made up your mind, though, so don’t feel obligated to continue to argue the case that no one will ever solve what you believe is unsolvable.
That is a terrifying stance. And, frankly, embarrassing.
“OpenAI admits AI hallucinations are mathematically inevitable, not just engineering flaws”: https://www.computerworld.com/article/4059383/openai-admits-ai-hallucinations-are-mathematically-inevitable-not-just-engineering-flaws.html
I, of course, disagree. Humans on the whole are ignorant and the average person is insufferably ill-informed. You are proving to be the exception. In general however, people suck at answering questions on subjects in which they’re not experts.
I have read that study, but I believe it to be flawed and I’m not alone. Please read the counter argument that hallucination is a structural phenomenon of estimation itself.
In other words, it is a problem that can be solved.
https://arxiv.org/abs/2509.21473?hl=en-US