> LLMs are here to stay. I’m increasingly confident that there aren’t any big breakthroughs coming by just throwing more data and computational power at LLMs
It seems true that basic data/compute scaling is showing diminishing returns, but there's still a lot of room for growth in reasoning/agentic models. It's not a coincidence that a bunch of OpenAI execs who left started working in that area.
I agree that this is the interesting space to watch, but it's my understanding that the reasoning/agentic models are not really LLMs alone, but more like sidecar algorithms that wrap around an LLM. I'm sure that we'll get some new gains here, but I think the fact that we see the departures from OpenAI indicate the LLMs alone are not the silver bullet here.
This isn't to say that LLMs won't play an important role, just that they are not enough on their own and we still need a new invention to get to the next level and, as far as I understand, we don't have that yet.
I would say also that even if there's stagnation in the capability of frontier LLM chatbots, we could see advancements in their application to specific industries like education. We're in a period of overhang in the economy where firms mostly have yet to accommodate LLM tools into their workflows (or lose out to new firms if their systems are too ossified to use them effectively).
100% agree. This sort of fine-tuned application is exactly the kind of work I think will be fruitful to work with students on. Haven’t played around with it much yet, but I have high hopes for what can be prototyped with tools like langchain. https://github.com/langchain-ai/langchain
Makes sense. I do LLM research and have been planning to get into the space and doing this kind of applied ML work (from the exam-prep side; selling to schools seems a bit opaque/Byzantine if I'm not already an insider). I'd be interested in talking more in DMs if you're interested.
I credit you frequently for the "engage yes, integrate maybe" approach I now promote. I'm curious how much time you spend, if any, explaining how these tools work? I feel like your students, being future engineers from Harvey Mudd, may be coming in with a better basic understanding of this than most? Or not?
I really don't spend a ton of time talking about all the details about how they work. I think that probabilistic next-token prediction algorithms, while a mouthful, actually does a pretty good job of getting 90% of the way there. LLMs are really sophisticated autocomplete. Once you have this understanding in place, then the space for useful applications begins to open up. Sort of like discovering that the hammer is good for nails, but not screws.
I think that Mudders likely have a better grasp of these than most, but I don't think you need an engineering or CS degree to understand how these work at the level that matters to use them wisely.
Thank you for this sensible, wise, grounded approach. It expresses pretty much how I'm approaching things as well. Somewhere in their education students need experiences using AI, but these should be carefully chosen to meet other pedagogical goals as well. And we need to make sure they don't replace important learning experiences..
Thank you, Anna. I’m glad to hear it resonated with your approach too. We need to cut through the extreme takes and find approaches grounded in our pedagogical values. There is a third way, and I’m trying to chart that path and walk it. Glad to be finding the community of many other like-minded folks like you!
> LLMs are here to stay. I’m increasingly confident that there aren’t any big breakthroughs coming by just throwing more data and computational power at LLMs
It seems true that basic data/compute scaling is showing diminishing returns, but there's still a lot of room for growth in reasoning/agentic models. It's not a coincidence that a bunch of OpenAI execs who left started working in that area.
I agree that this is the interesting space to watch, but it's my understanding that the reasoning/agentic models are not really LLMs alone, but more like sidecar algorithms that wrap around an LLM. I'm sure that we'll get some new gains here, but I think the fact that we see the departures from OpenAI indicate the LLMs alone are not the silver bullet here.
This isn't to say that LLMs won't play an important role, just that they are not enough on their own and we still need a new invention to get to the next level and, as far as I understand, we don't have that yet.
That's definitely fair.
I would say also that even if there's stagnation in the capability of frontier LLM chatbots, we could see advancements in their application to specific industries like education. We're in a period of overhang in the economy where firms mostly have yet to accommodate LLM tools into their workflows (or lose out to new firms if their systems are too ossified to use them effectively).
100% agree. This sort of fine-tuned application is exactly the kind of work I think will be fruitful to work with students on. Haven’t played around with it much yet, but I have high hopes for what can be prototyped with tools like langchain. https://github.com/langchain-ai/langchain
Makes sense. I do LLM research and have been planning to get into the space and doing this kind of applied ML work (from the exam-prep side; selling to schools seems a bit opaque/Byzantine if I'm not already an insider). I'd be interested in talking more in DMs if you're interested.
I credit you frequently for the "engage yes, integrate maybe" approach I now promote. I'm curious how much time you spend, if any, explaining how these tools work? I feel like your students, being future engineers from Harvey Mudd, may be coming in with a better basic understanding of this than most? Or not?
Thanks Ben, glad it's been a helpful approach!
I really don't spend a ton of time talking about all the details about how they work. I think that probabilistic next-token prediction algorithms, while a mouthful, actually does a pretty good job of getting 90% of the way there. LLMs are really sophisticated autocomplete. Once you have this understanding in place, then the space for useful applications begins to open up. Sort of like discovering that the hammer is good for nails, but not screws.
I think that Mudders likely have a better grasp of these than most, but I don't think you need an engineering or CS degree to understand how these work at the level that matters to use them wisely.
Thank you for this sensible, wise, grounded approach. It expresses pretty much how I'm approaching things as well. Somewhere in their education students need experiences using AI, but these should be carefully chosen to meet other pedagogical goals as well. And we need to make sure they don't replace important learning experiences..
Thank you, Anna. I’m glad to hear it resonated with your approach too. We need to cut through the extreme takes and find approaches grounded in our pedagogical values. There is a third way, and I’m trying to chart that path and walk it. Glad to be finding the community of many other like-minded folks like you!