LLMs Are Not Intelligent

A critical point to help you understand and work with AI effectively

Thank you for being here. As always, these essays are free and publicly available without a paywall. If you can, please support my writing by becoming a patron via a paid subscription.

Photo by Lucas Santos on Unsplash

We need some definitions in our conversation about artificial intelligence. As a category of technology, AI refers to a wide array of methods and systems. As we look back at the explosion of new ideas and technologies over the past few years in the space, we need to do some housecleaning.

But there is one question that dwarfs all others at the heart of AI discourse: what is intelligence?

Before we can ask whether a machine can exhibit it, we've got to have at least a working definition of what it is. Understanding the nature of intelligence can help us wade through the hype around AI and keep our focus on the pressing questions about the societal impact of our current AI-powered tools.

LLMs and The Intelligence Age

Unsurprisingly, definitions remain a topic of debate and often remain vague. For example, if you read between the lines of his recent essay "The Intelligence Age," Sam Altman seems to define intelligence as the capacity to learn from data. The advances in AI we see today, in Altman's mind, are directly tied to one thing: the success of deep learning.

That’s really it; humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data). To a shocking degree of precision, the more compute and data available, the better it gets at helping people solve hard problems. I find that no matter how much time I spend thinking about this, I can never really internalize how consequential it is.

It's interesting to see the shift in messaging in this essay compared to earlier ones. I sense that the goalposts are moving. The vision for AGI (i.e., artificial general intelligence) from OpenAI in the early months of 2023 was squarely focused on a decidedly different sort of thing, at least by my read. The 2023 post seems to conjure the AI systems of science fiction that operate autonomously without human direction or control.

And yet, in reading Sam's Intelligence Age piece from a few weeks ago, the tack seems to have changed to one much more aligned with a vision of AI co-pilots that help people solve hard problems, rather than solving them problems on their own independent from human input.

I am convinced that our recent advances in generative AI are consequential, even without a massive, magical jump in performance or capability. I expect that the progress will continue to be largely incremental, in the same vein as the new releases we have been seeing in recent months. Companies will leverage LLMs and other generative AI tools like them to use your computer or create audio podcast summaries of the information you feed them. And yet, while these advances may be incremental, they will have far-reaching consequences as generative AI continues to seep into the tools we use every day.

François Chollet on the True Nature of Intelligence

The initial idea for this essay was prompted by a podcast conversation I listened to last week with software engineer and AI researcher

François Chollet

on

Sean Carroll

’s Mindscape podcast. In their conversation from June of 2024, Chollet makes his case for what intelligence is and why LLMs don't have it. Their conversation can help us to understand what LLMs are and their potential uses and dangers across a range of different applications. It's a useful point of comparison to Altman's claims of the intelligence age.

One of the reasons the conversation is so valuable is that Chollet spends time exploring the definitions of terms that are so often assumed or left undefined. With his extensive background in machine learning and AI, Chollet is one of the best people in the world to demystify how LLMs work and clarify what they do vs. what they seem to do.

Chollet argues that LLMs are not intelligent because they can’t generalize. In his widely read and cited 2019 paper “On the Measure of Intelligence,” Chollet quotes Legg and Hutter’s summarized definition: “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” This definition, Chollet says, describes two main components of intelligence:

1. Task-specific skill

2. The ability to generalize

I'm not so sure that Altman would disagree with Chollet's definition, but as a storyteller and CEO trying to raise money for new chips and data centers, the fuzzier and more romantic the definition the better. In an early 2023 blog post designed to quell our fears about OpenAI's quest for AGI, Sam defines AGI as "AI systems that are generally smarter than humans." These two definitions may not be mutually exclusive, but one is a lot easier to get your hands around and test than the other.

After listening to Chollet's interview, I'm pretty convinced about one thing: LLMs are—at least on their own—a dead end to the sort of superhuman intelligence that we think about when we hear the term AGI. In his essay, Sam argues that the intelligence age is imminent "It is possible that we will have superintelligence in a few thousand days (!); it may take longer, but I’m confident we’ll get there." Whether we can get to AGI or not is still an open question and given the recent changes and departures at OpenAI it seems reasonable to have some doubts about how close we are. We'll certainly get "there," but that destination depends a lot on our definitions.

Memorization is not intelligence

Why are LLMs a dead end? In short, memorization does not equal intelligence. Chollet explains that LLMs amaze us because of their ability to plausibly navigate their training data—that is many orders of magnitudes larger than any we've ever interacted with before—in a flexible and natural way. Before interacting with ChatGPT, the closest LLM-like experience was probably with a curious, highly intelligent, and widely-read friend with a knack for memorization. But they just can't compete with an LLM that has been trained on the entire internet.

But memorization and intelligence are not the same thing. We often conflate these even in humans, but the ability to memorize and accurately recall a large amount of information is not the same as being able to generalize that memorized information to a skill that you've never practiced before.

Mathematically, we can think of LLMs and other machine learning algorithms as sophisticated curve-fitting operations. The underlying training data can be represented in space with many dimensions and then we can train a model to approximate that function. Then, after fitting a function to the data, we can evaluate it at different points. In the case of LLMs, this evaluation process is where we inject the user prompt and get the output.

But, Chollet argues, this process of prompting the system is fundamentally function interpolation. To rephrase in plain English, LLMs operate within the confines of their training data. While they may generate a response that we've never seen before, it is simply doing a pattern-matching operation and combining various aspects of its training data. It is, in essence, fetching a response that weaves together different parts of memorized data.

Revisiting our definition of intelligence, memorization clearly fails the test. While memorization can certainly help you to develop a task-specific skill (part 1), it does not help you to approach arbitrary skills that exist outside of what you've memorized (part 2).

This is not to say that LLMs are useless. We are just now scratching the surface of the potential applications of this technology. If we hop off the LLM hype train that's en route to the mirage of superintelligence, I suspect we are likely to find lots of creative applications of this technology. My hunch is that the most fruitful applications of generative AI and of LLMs will be found in partnership with human intelligence.

You've likely already found ways to do this and I think there's more coming. In my own workflow, I'm using generative AI to help me better explore information from different angles. Using it to summarize and highlight themes from feedback and provide suggestions for action can help equip you to make changes with your team or class. Imagine an AI system that is fine-tuned to help suggest active learning activities to try out in your class, curated based on research-backed studies that you may not even be aware of.

Using LLMs in this way leverages their ability to access some kinds of data much more efficiently than we can. Often the problem is not that we don't want to make a better decision or apply a best practice—we just don't know about it or don't remember it. LLM-powered applications can help in these situations.

The key to using LLMs well is to realize what they are and how they work. They are not intelligent. But that doesn't mean they're not useful.

As we use them, let's not forget Sam's claim that this new technology has us "on the cusp of the next leap in prosperity." More prosperity? Perhaps. But important questions remain:

• Prosperity for whom?

• At what cost?

• Toward what ends?

And perhaps most importantly: to what problem is more prosperity the solution?

Got a comment? I’d love to hear it.

Reading Recommendations

First of all, the podcast conversation with François is well worth your time. I highly recommend you give it a listen.

I also recommend giving Chollet’s 2019 paper a read and playing with the ARC prize website. The logic games are fun!

In the essay he posted yesterday,

Ethan Mollick

hits home on an important point: the impact of AI is here and will continue to unroll even without the arrival of superintelligence.

Organizations need to move beyond viewing AI deployment as purely a technical challenge. Instead, they must consider the human impact of these technologies. Long before AIs achieve human-level performance, their impact on work and society will be profound and far-reaching. The examples I showed —from construction site monitoring to virtual avatars—are just the beginning. The urgent task before us is ensuring these transformations enhance rather than diminish human potential, creating workplaces where technology serves to elevate human capability rather than replace it. The decisions we make now, in these early days of AI integration, will shape not just the future of work, but the future of human agency in an AI-augmented world.

This interview between

David Epstein

and journalist Amanda Ripley on ways to have better conflict was a great read. I particularly liked this idea about “looping for understanding.” Likely to be particularly timely reading given the election today.

The steps are: listen to what's most important to the other person — not most interesting to you, but what seems most important to them — then distill it into your own most elegant language, and then play it back. And then — and this is the one that's easy to forget — check if you got it right. It's almost like what you mentioned with Selena, when she was saying: Help me understand this. For example: “It sounds like you're saying that this election makes you feel profound despair, and most of all you wish it were over and you could just come back in a month. Is that right?” And just doing that all of the sudden reorders the universe, because people can sense that you're trying. And then they start to correct you, or add on. They also get to hear themselves, which is really, really important. 

---

The Book Nook

I’m finally digging into Hannah Arendt's The Human Condition. Just a few pages in now, but I’m looking forward to engaging with her thinking.

---

The Professor Is In

Next week I’m offering a workshop at Mudd to help students prototype a portfolio website. It’s going to be based on the template I put together for E155 this fall in Quarto. Hoping that it will be a helpful exercise and tool for students in their professional journeys.

I’ll plan to share an update here along with a link to the template if you want to follow along and create your own.

---

Leisure Line

Here are the Brake household 2024 jack-o-lanterns with a cameo appearance from #1’s latest favorite snowmobile Lego set.

---

Still Life

I spent last Friday and Saturday in Palm Springs for the Harvey Mudd Board of Trustees retreat. It was a great time with colleagues and the board and got to tour Sunnylands while we were there.