Excellent piece, Josh. I am sure I was channeling you talking about prototyping as I developed a plan to have an LLM augment an online discussion board in my history of higher education class. The experiment is designed to see if narrowly tailored responses by an LLM encourages students to write more in their discussion posts, which will give their classmates more to respond to, which will help everyone, including me, prepare for class. I kept reminding myself, this is just a first try...doesn't need to be perfect and it may not work. And, since the point is for the class to learn something about LLMs, it will succeed even if it fails to confirm the hypothesis.
Rob, this is great! I'm so glad to hear about this experiment, thank you for sharing. I love that you've got a clear hypothesis and a way to evaluate how it goes. I'll be curious to hear how it goes and what you learn.
I especially appreciate the trust angle on this — I feel like there’s always more excitement to take academic/pedagogical freedom more seriously when it’s clear professors trust you and are excited about seeing what you learn. My friend Sheon wrote this great profile of Manuel Blum which has a similar read on why he was so effective as a mentor: besides being incredibly encouraging, he seriously wanted to learn something new from his students! https://www.technologyreview.com/2023/10/24/1081478/manuel-blum-theoretical-computer-science-turing-award-academic-advisor/
Also, I don’t know if you’ve ever spoken with James Kreines from the CMC philosophy department, but he’s been thinking about some of these pedagogical questions and playing around with the tools for a while.
Thanks Daniel. Yes, the idea of trust is becoming more and more central to my pedagogy these days. I just don't see a way in which investing in building trust isn't worthwhile. Feels like there are so many benefits for students and for me as an instructor and that it is a natural way to help align our interests and promote learning.
Thanks for the link to the Blum profile, I will take a look.
And no, I haven't connected with James Kreines over at CMC but will look him up. Thanks for the rec.
Also, please come say hi next time you're on campus. Maybe we can record something together and/or find other interesting ways to have current students connect with you.
Great post, Josh. I think your approach resonates with mine, and I'd be interested in seeing more detail. How about an exchange of syllabusses? Anyway, it looks like the West Coast and the East Coast will be in accord at points this fall semester! Enjoy the last hours of summer.
Thanks Mark, would be happy to share syllabi. Mine is public on the web at https://hmc-e155.github.io/syllabus/. Likely very different than yours given the different type of class, but might be interesting nonetheless. I'd be curious to hear your feedback and suggestions coming from a very different field. Would love to see your syllabus as well so feel free to send it over. You know where to find me :)
I often think back to the tasks that I had to slog through to complete projects when I was in school. At the time, they were never fun, and many of the chores I had to do can now be automated. Yet, having to undertake those tasks proved to be priceless learning experiences that have underpinned my ability to perform subsequent projects. For example, being forced to create my own datasets from numerous original sources was so tedious and time consuming, yet it forced me to really understand all the nuances in my data. I believe this is an understanding that too many analysts who are simply provided with readily available datasets lack, which dooms their reports. I think about how to communicate to students or underlings the necessity of going through the process to motivate them to muddle through, rather than taking some easy way out. I'm not so sure I can provide a compelling case.
Note: I'm not an academic - considered that pathway, decided an MA (Philosophy) was enough for me. Instead, I contribute the thoughts of a friend of mine from college, who now post-English PhD teaches freshman English composition at a major West Coast university.
The students who are completing their work using ChatGPT, in my friends classes - the ones she catches, at least - are overwhelmingly students laboring under enormous and mounting debts. They are often international students, who are paying many multiples, just for their basic courses, of what domestic students are, never mind room and board. Her course is just an obstacle to them, a box to tick. Graduating is all that matters. She is uncertain how to sell the "value of learning" and "learning to think" to people in their situation.
I bring this up to ask for reflection on the realities of your students situations. How realistic is to ask them to value "the education/learning to think" when they face decades of debt repayments, an increasingly uncertain job market (full disclosure: I am one of the people working to replace low-level human cognitive labor with AI), and an overwhelmingly imperative just to get the degree? +
I was immensely privileged: I graduated a decade and a half ago without student debt, which was a very good thing as I struggled for half a decade post-graduation to find a career. I had the luxury of treating university as an opportunity for learning, for expanding my mind, learning how to write to think. I agree with my friend: I don't know what to tell students who have been told (a) this is the only way to ever get a good job, and (b) you must take on debt up to your eyeballs to do it, compromising your entire first two decades of start in life.
Not defending cheating, at all, just asking for some respect for the situation of current students.
A lot of your post hits home, including “Trying to make your assignments more AI-proof is a waste of time. What you should do instead is to help get your students engaged in the work they are doing. To tell them why it will be valuable and why it's worth their time. To tell them that your goal in giving them feedback, and yes, grades when you give them, is to help them grow and reflect on how they might further develop themselves to create the type of life that will help them to flourish along with the people around them.”
My academic department (languages & literature) met for several hours yesterday. AI use and response was one of the topics. But I’m starting to feel that we (my department) are spending too much effort in the “AI response/prevention” category and not enough in the “revising grading approaches” category. I’m wondering if the grading part needs adjustment almost before the “responding to AI by valuing process more” does.
Excellent piece, Josh. I am sure I was channeling you talking about prototyping as I developed a plan to have an LLM augment an online discussion board in my history of higher education class. The experiment is designed to see if narrowly tailored responses by an LLM encourages students to write more in their discussion posts, which will give their classmates more to respond to, which will help everyone, including me, prepare for class. I kept reminding myself, this is just a first try...doesn't need to be perfect and it may not work. And, since the point is for the class to learn something about LLMs, it will succeed even if it fails to confirm the hypothesis.
Rob, this is great! I'm so glad to hear about this experiment, thank you for sharing. I love that you've got a clear hypothesis and a way to evaluate how it goes. I'll be curious to hear how it goes and what you learn.
I especially appreciate the trust angle on this — I feel like there’s always more excitement to take academic/pedagogical freedom more seriously when it’s clear professors trust you and are excited about seeing what you learn. My friend Sheon wrote this great profile of Manuel Blum which has a similar read on why he was so effective as a mentor: besides being incredibly encouraging, he seriously wanted to learn something new from his students! https://www.technologyreview.com/2023/10/24/1081478/manuel-blum-theoretical-computer-science-turing-award-academic-advisor/
Also, I don’t know if you’ve ever spoken with James Kreines from the CMC philosophy department, but he’s been thinking about some of these pedagogical questions and playing around with the tools for a while.
Thanks Daniel. Yes, the idea of trust is becoming more and more central to my pedagogy these days. I just don't see a way in which investing in building trust isn't worthwhile. Feels like there are so many benefits for students and for me as an instructor and that it is a natural way to help align our interests and promote learning.
Thanks for the link to the Blum profile, I will take a look.
And no, I haven't connected with James Kreines over at CMC but will look him up. Thanks for the rec.
Also, please come say hi next time you're on campus. Maybe we can record something together and/or find other interesting ways to have current students connect with you.
That would be lovely! I hope I’ll have the chance to visit LA sometime this year, and I’ll absolutely let you know and stop by if I make it.
Sounds great—do let me know!
Great post, Josh. I think your approach resonates with mine, and I'd be interested in seeing more detail. How about an exchange of syllabusses? Anyway, it looks like the West Coast and the East Coast will be in accord at points this fall semester! Enjoy the last hours of summer.
Thanks Mark, would be happy to share syllabi. Mine is public on the web at https://hmc-e155.github.io/syllabus/. Likely very different than yours given the different type of class, but might be interesting nonetheless. I'd be curious to hear your feedback and suggestions coming from a very different field. Would love to see your syllabus as well so feel free to send it over. You know where to find me :)
I often think back to the tasks that I had to slog through to complete projects when I was in school. At the time, they were never fun, and many of the chores I had to do can now be automated. Yet, having to undertake those tasks proved to be priceless learning experiences that have underpinned my ability to perform subsequent projects. For example, being forced to create my own datasets from numerous original sources was so tedious and time consuming, yet it forced me to really understand all the nuances in my data. I believe this is an understanding that too many analysts who are simply provided with readily available datasets lack, which dooms their reports. I think about how to communicate to students or underlings the necessity of going through the process to motivate them to muddle through, rather than taking some easy way out. I'm not so sure I can provide a compelling case.
Note: I'm not an academic - considered that pathway, decided an MA (Philosophy) was enough for me. Instead, I contribute the thoughts of a friend of mine from college, who now post-English PhD teaches freshman English composition at a major West Coast university.
The students who are completing their work using ChatGPT, in my friends classes - the ones she catches, at least - are overwhelmingly students laboring under enormous and mounting debts. They are often international students, who are paying many multiples, just for their basic courses, of what domestic students are, never mind room and board. Her course is just an obstacle to them, a box to tick. Graduating is all that matters. She is uncertain how to sell the "value of learning" and "learning to think" to people in their situation.
I bring this up to ask for reflection on the realities of your students situations. How realistic is to ask them to value "the education/learning to think" when they face decades of debt repayments, an increasingly uncertain job market (full disclosure: I am one of the people working to replace low-level human cognitive labor with AI), and an overwhelmingly imperative just to get the degree? +
I was immensely privileged: I graduated a decade and a half ago without student debt, which was a very good thing as I struggled for half a decade post-graduation to find a career. I had the luxury of treating university as an opportunity for learning, for expanding my mind, learning how to write to think. I agree with my friend: I don't know what to tell students who have been told (a) this is the only way to ever get a good job, and (b) you must take on debt up to your eyeballs to do it, compromising your entire first two decades of start in life.
Not defending cheating, at all, just asking for some respect for the situation of current students.
A lot of your post hits home, including “Trying to make your assignments more AI-proof is a waste of time. What you should do instead is to help get your students engaged in the work they are doing. To tell them why it will be valuable and why it's worth their time. To tell them that your goal in giving them feedback, and yes, grades when you give them, is to help them grow and reflect on how they might further develop themselves to create the type of life that will help them to flourish along with the people around them.”
My academic department (languages & literature) met for several hours yesterday. AI use and response was one of the topics. But I’m starting to feel that we (my department) are spending too much effort in the “AI response/prevention” category and not enough in the “revising grading approaches” category. I’m wondering if the grading part needs adjustment almost before the “responding to AI by valuing process more” does.