AI Tutors Can't Solve Bloom's Two Sigma Problem
The personal cannot be mechanized
Thank you for being here. As always, these essays are free and publicly available without a paywall. If you can, please consider supporting my writing by becoming a patron via a paid subscription.
Efficiency and immediacy. The idols of our time. We want it without effort and we want it now. In educational spaces, the problem is that the most efficient path is often the one with the least amount of learning. Learning is the result of friction.
There are many reasons we’re conditioned this way, but technological innovation plays a large part. Technology gives us power over the world, from controlling the temperature in our houses at the push of a button to rapidly cooking food in our microwaves, to beaming our faces across the globe from the smartphones in our pockets.
But for all the new power and comfort that technology has provided us, it's not clear to me that technology has been forming us in positive ways.
Whether you think AI-powered chatbots are the future of education is fundamentally linked to what you think teachers do. Can the work of a creature be fully outsourced to a machine?
The Absent-Minded Professor is a reader-supported guide to human flourishing in a technology-saturated world. The best way to support my work is by becoming a paid subscriber and sharing it with others.
What are we cultivating?
Virtues and vices provide a helpful lens for assessing our formation. What if we asked not what technology can do for us, but to what degree will it foster virtue? Does my smartphone help me cultivate patience, courage, truthfulness, and modesty? Or does it create fertile ground for jealousy, envy, hate, vanity, and self-indulgence? While we often limit our definition of technology to electronic devices, it's a much broader category that could encompass devices like dishwashers, automobiles, and even books.
I hope that we can think about the influence of generative AI through this lens of virtue and vice cultivation. As a professor who feels a calling to form, shape, and train the next generation of engineers, I am particularly concerned about how generative AI will shape the future of education.
Here's the thing: generative AI is going to revolutionize learning. In so many ways, it already has, simply by dominating much of the conversation in educational spaces over the last several years. What gives me hope is that in this period of technological disruption, we have an opportunity to return to the drawing board and think broadly about what we want the future of education to look like.
In my view, there are two main options on offer: we can either use technology to replace or to augment the work that teachers are doing. This is not always a clear distinction in practice, but in broad strokes, this is a helpful distinction to be thinking through.
For some, this distinction breeds fear. We've all heard the rhetoric many times over now about how generative AI is uniquely positioned to disrupt white-collar work, something that is relatively unique where most technological innovation primarily impacts blue-collar occupations.
But I see a silver lining: generative AI cannot replace teachers. Let me explain. While technology can replace some aspects of the work of teaching, it is fundamentally unable to replace everything that teachers do. We can imagine automating our lesson planning and grading, but no technology, generative AI included, can fully reproduce the craft of teaching. It's like trying to describe a sphere with only two dimensions. The best you can do is the low-dimensional representation of a circle.
Teaching as a creaturely task
Technology cannot replace teaching because teaching is a creaturely task. I realize that I'm making an anthropological and philosophical claim here. But I think that if you’ve ever been in the presence of a great teacher, you’ll have a sense of what I’m talking about here. This idea is rooted in the experience that there is something ineffable to the human experience, something that points to something beyond our flesh and blood and the firing of our neurons.
What has made our current moment particularly challenging in this respect is that LLMs are excellent deceivers. Because they are trained on language, our language, they sound like us. So convincingly sometimes it is hard to hold fast to the idea that there is not a conscious or at least sentient being under the hood. Just ask Blake Lemoine. But as difficult as it is to hold fast to this idea, we can and we must. While these systems can give the illusion of empathy or encouragement, it's just that—an illusion. There is no there there.
What is learning?
This connects to education because learning is not a problem, primarily, of information download. It's not just about memorizing more information or building skills faster. More knowledge and expertise are not in and of themselves bad, but they're not the point of the enterprise.
Our modern obsession with efficiency and immediacy coupled with our societal and economic systems is the perfect storm. It has been and will continue to create significant issues in learning spaces. Friction and inefficiency is a fundamental part of what it means to learn. It takes energy and struggle to rewire our brains to learn something new. It takes failure.
The question we ultimately get back to is one of purpose. What is education for? We've got to be clear on the ends before we worry about the means.
I've been workshopping a thesis in my mind over the past few months: the future of education is personal, not personalized.
At first glance, you may think these are the same thing. But they’re not.
A personal education is one where the ends and means are concerned with persons. The goal is to cultivate a person through relationships with other persons. Personalization, on the other hand, is the appearance of personhood without its substance. Personal is the email from an old friend. Personalized is the form email from someone who knows your name, but nothing else.
How do AI chatbots play in the world of personal vs. personalized learning? Let's look at generative AI-powered tutor chatbots like Khanmigo. Perhaps more than any other AI-powered educational technology, GPT-powered tools like Khanmigo are getting a lot of airtime and attention.
AI-powered tutors are an excellent example of the difference between personal and personalized. The idea behind these chatbots is well-intentioned. We've known for over forty years now that 1-on-1 tutoring is an extremely powerful tool for helping students to learn. In a landmark 1984 paper, educational psychologist Benjamin Bloom and a few of his graduate students demonstrated that 1-on-1 tutoring significantly improved student achievement, bringing the average student receiving 1-on-1 tutoring nearly two standard deviations above the average student in a conventional classroom setting.
In case it's been a while since your last stats class, this is a significant effect. This means the average student receiving 1-on-1 tutoring performed above 98% of the conventional group. Huge. Unfortunately, as we know, 1-on-1 tutoring is exactly the sort of intervention that is very costly and therefore inherently difficult to scale. Ever since, Bloom's Two Sigma problem has been a holy grail of sorts in educational technology as ventures seek to democratize access to 1-on-1 tutoring using technology.
But, as with so many technological solutions, the problem becomes a variation on the law of the instrument: to the one with the hammer, everything begins to look like a nail.
As we consider the role computerized tutors have to play in pedagogy, we need to take a step back and ask some questions. Questions like:
Why exactly is 1-on-1 tutoring so successful?
What is a tutor doing when they work with students?
What part, if any, of tutoring is a human uniquely qualified to do?
How well can a human tutor be emulated by an AI-powered one?
I was reminded of this in a post I ran across from
who writes the excellent Substack . This week he posted a video on his LinkedIn where he and a colleague annotate the work a teacher is doing in the classroom as if they were sports play-by-play sports commentators.
The point of Dan’s video is an echo of the drum he has been beating for a long time now: what teachers do in the classroom is way more sophisticated than it might look at first glance.
I would be surprised if Khanmigo doesn't have a positive impact on student learning, at least on some measures. But my hunch is that the impact of AI chatbots on education will be similar to the impact of massively open online courses (MOOCs): great for democratizing access to educational content, but ineffective at solving the stickiest problems of learning.
Fundamentally, this is because only a human can see students and engage with them in the context of a relationship; to encourage them to keep going even when they encounter struggle and want to give up. The world's most advanced personalized generative AI chatbot is fundamentally unable to replace the personal connection offered in the relationship between the teacher and student. ChatGPT or Khanmigo might be able to give me platitudes or encouraging messages to urge me to keep going even when I get a wrong answer, but it’s not genuine. This type of encouragement is fake.
The grand vision of AI chatbots replacing teacher-student relationships is not a vision of the world that I'm particularly excited about.
This isn't to say that I don't think generative AI has applications that could improve education. I've seen it in my own work with students: having access to an LLM in my pocket is helpful when I need to refresh on a particular topic and then understand how to share that information with my students. Just last week I used ChatGPT to help me refresh myself on some details about a circuit design problem to help some students. However, their experience of the LLM mediated through a relationship with an educator they trust is categorically different than having that same interaction with the LLM directly.
In developing new technology we always have choices. Choices about what we build, how we build, and why we build. To solve any problem, we've got to start by asking the right question. As I'm fond of saying to my students, the right answer to the wrong question is the wrong answer.
Recommended Reading
I enjoyed this post from
on creativity and craft.
This piece from
is a great analysis of Paul Graham’s singular style.
The Book Nook
Was reading a piece about AI this last weekend when I stumbled again on the recommendation to read C.S. Lewis’s The Abolition of Man. It’s a short (~40 page) read and one with some wisdom to speak to our current moment. A few quotes that resonated with me.
The task of the modern educator is not to cut down jungles but to irrigate deserts. The right defence against false sentiments is to inculcate just sentiments. By starving the sensibility of our pupils we only make them easier prey to the propagandist when he comes. For famished nature will be avenged and a hard heart is no infallible protection against a soft head.
St Augustine defines virtue as ordo amoris, the ordinate condition of the affections in which every object is accorded that kind of degree of love which is appropriate to it. Aristotle says that the aim of education is to make the pupil like and dislike what he ought. When the age for reflective thought comes, the pupil who has been thus trained in 'Ordinate affections' or 'just sentiments' will easily find the first principles in Ethics; but to the corrupt man they will never be visible at all and he can make no progress in that science.
This conception in all its forms, Platonic, Aristotelian, Stoic, Christian, and Oriental alike, I shall henceforth refer to for brevity simply as the Tao. Some of the accounts of it which I have quoted will seem, perhaps, to many of you merely quaint or even magical. But what is common to them all is something we cannot neglect. It is the doctrine of objective value, the belief that certain attitudes are really true, and others really false, to the kind of thing the universe is and the kind of things we are. Those who know the Tao can hold that to call children delightful or old men venerable is not simply to record a psychological fact about our own parental or filial emotions at the moment, but to recognize a quality which demands a certain response from us whether we make it or not.
The Professor Is In
It’s launch week this week. On Saturday students will load up on the buses first thing in the morning and ship out to Dana Point to launch their robots in the bay. I’m sure I’ll have pictures to share next week!
Leisure Line
Several baking adventures over the weekend, but none quite as delicious as this. Easy to boot. Gift link to the recipe from NYTimes Cooking in case you’re interested.
Still Life
These guys have been all over the honeysuckle in our yard and at war with some birds. I imagine they are trying to get at some eggs in the nest without getting their eyes pecked out. This picture of them just chilling out was too much! I can imagine that
will get a kick out of this.
In UK the people who send their kids to feel paying schools,often at the cost of struggle and going without,do so because they know that Education is about a lot more than cramming ones brain with innumerable facts. In vain will their local State Comprehensive flourish charts showing the high number of A* pupils it produced every year,all candidates for university,at least. I find it a bit sad that all our politicians see "social mobility" as having a well paid job thanks to those A*'s. I think its such a narrow definition. I enjoy classical music,go to the theatre,travel if I've ever got any money,and cook meals where all the ingredients are mixed up together not on the plate in separate dollops(think dire 1950s British home cooking) I regard this as SOCIAL MOBILITY since I was brought up in a family that self-barred from all this with the mantra "People Like Us Don't Do Things Like That". How,I hated being People Like Us,I wanted to be People Like Them.
The writer LYNSEY HANLEY from a similar background writes excellently about this.
Those squirrels! Such a great photo. I do absolutely get a kick out of this. (Also hope the eggs stayed safe.) Thanks for sharing!