How LLMs Fool Us Into Believing They Are Smart
During the first decade of the 20th century, a horse named Hans drew worldwide attention as the first thinking animal. Hans solved calculations and performed other amazing feats by tapping numbers or letters with his hoof to answer questions from audience members.
Apart from a few skeptics, experts were convinced that the horse was able to understand and reason like a human. And it certainly appeared that way.
Then, Oscar Pfungst, a biologist and psychologist, found that the horse couldn’t answer the question if the questioning person didn’t know the answer. How peculiar! It turned out Hans was indeed clever, but not in the way people had thought.
The horse was in fact an outstanding observer. It had learned to tell from reading body language and facial expressions when he had tapped or was about to tap the correct number or letter, after which he would receive a reward. To get it right, Clever Hans didn’t have to understand a word of what was being said. It was enough to fool the world into believing he was smart.
The year is 2023. We are much cleverer now. We have spaceships, smart phones, and wireless Internet. We would never fall for something like that today, would we?
If you haven’t caught onto it yet, the story of Clever Hans is surprisingly analogous for how large language models trick us into believing they’re smart. It serves as a modern parable about a kind of gullibility that strikes even the most intelligent among us — the smarter you are, the stronger the effect.
Many experts are convinced LLMs are intelligent. Prominent people with exceptional track-records, like AI Researcher Geoffrey Hinton, who pioneered the technology for current systems such as ChatGPT. In an interview with 60 Minutes, he said: “we’re moving into a period when for the first time ever we may have things more intelligent than us”. Or take Blaise Agüera y Arcas, VP at Google Research, and Peter Norvig, former Director of Research at Google, who published a piece last week arguing that artificial general intelligence is already here.
These are not your average Joe’s, and I would never dare to challenge them if the evidence of the contrary wasn’t so glaringly obvious. So, let’s channel our inner Oscar Pfungst and examine.
We know by now that large language models predict the likelihood of the next word in a sequence. Any response you get is a best-guess based on statistical relevance. They achieve great accuracy, because they are trained on huge amounts of text data and optimized through reinforcement learning with human feedback (RLHF).
This ‘truth by proximity’-approach that sits at the heart of this technology has proven to be incredibly potent, but it also introduced the world to an entirely new phenomenon: hallucinations. Basically, these models don’t know what they don’t know. They will ‘lie’ just as easily as ‘tell the truth’, which to this day remains an unsolved problem.
It turns out LLMs can’t really reason, either. They can approximate something that looks like reasoning, but the process is flawed. This is best illustrated by a recent paper The Reverse Curse, showing that if a model is trained on “A is B”, it will not automatically generalize that “B is A”.
In the paper’s abstract, it reads:
“For instance, if a model is trained on “Olaf Scholz was the ninth Chancellor of Germany”, it will not automatically be able to answer the question, “Who was the ninth Chancellor of Germany?”. Moreover, the likelihood of the correct answer (“Olaf Scholz”) will not be higher than for a random name. Thus, models exhibit a basic failure of logical deduction and do not generalize a prevalent pattern in their training set (i.e. if “A is B” occurs, “B is A” is more likely to occur).”
A third example brought to my attention by
involves math. In a recent post, he shared a table from a paper assessing multi-digit arithmetic in LLMs — it something they can’t seem to do, but this doesn’t stop them from happily providing us with the wrong answers as if they knew.As a frame of reference, Marcus was so kind to add column for ‘Calculator’, to which I was so kind to add another for our famous horse ‘Clever Hans’.
Jokes aside, the point is not to say LLMs aren’t useful or competent — in many ways, they are. What they aren’t, however, is smart. And we certainly haven’t reached general artificial intelligence, by any stretch of the imagination.
It’s funny, actually, how close the similarities are between the behavior of LLMs and that of Clever Hans. The horse learned, through practice, to read body language and micro-expressions. LLMs are great observers, too, but instead they look at actual language.
Just like Clever Hans, these models can give off the impression they understand and reason in the same way we do. They appear smart even to the cleverest among us, because the more you know, the more impressed you are by what it gets right.
LLMs are also incentivised with rewards. Anybody that knows a thing or two about RLHF knows that this process is designed to make models more ‘aligned’ with ‘human preferences’. In simple terms, LLMs are being rewarded for appearing smart, just like Clever Hans was.
Now, you might think, does it really matter? Sure, these machines operate and ‘think’ differently to us, but who cares if the internal processes are different — if it gets it right, it gets it right.
To that I say, even if machines in the coming decade will reach a level of posturing that makes it for impossible most people distinguish from the real intelligence, it still wouldn’t make them smart. It would be disingenuous to lower the bar of what it means to think and feel, just because there is a linguistically fluent machine that can convince us it is doing either of those things.
We don’t call calculators smart, either. And no serious person believes horses speak our language and can perform arithmetic, just because they can tap their hoofs at the correct number or letter.
PS. It looks like AI hasn’t figured out how many legs a horse has 👀
Join the conversation 💬
Leave a comment with your thoughts. Would you like to have your own horse to perform arithmetic in front of a big audience?
ncG1vNJzZmiipae0prrGq5ivnaOpsqq6jaysm6uklrCsesKopGioX6m1pnnCpZyvnaJitaK60macn56VmMFutM6wZKWknah6p7vOpQ%3D%3D