Love seeing examples like this, Mike. How did you all go about building the Jay Gatsby bot? How would a teacher think about making one like this for other characters in other books?
Building custom bots for these types of activities is more art than science -- and it's probably the subject for an entirely separate post (or workshop series).
But a couple key thoughts:
1. What do you want the student to demonstrate in the chat? Build the bot to behave in a way that creates those opportunities.
For example, for Holden Caulfield, you may have to toggle the depression a bit. I want my students to be tested but I don't want the bot spiraling out of control. For Gatsby - toggle the grandiosity and abstractions. I want to create opportunities for the student to ask incisive questions about personal motivation that will force the bot to "face up" a bit but also not lose its "personality." For Romeo, toggle the wistfulness and self-pity.
This is also highly relevant in building simulations and role-playing scenarios. For instance, I built a simulation for a political science class where students negotiate with fellow legislators and lobbyists. In that case, the question is "how do I want the other stakeholders to engage with the student so that the student has a fair shot at demonstrating their chops?"
Again, this is more art than science. Perhaps it is time to write up a guidebook! But I perceive that educators are not viewing chatbot interactions as potential tests, which is the first step.
It is an art! I do think about some of the base skills that teachers would want/need: what information do you need to provide the bot to have as background information? is that available on the web? do you need to include an attachment? what are the fair use rules for more recent publications? What are additional texts or reference documents that would be helpful to include (for which having clarity on the student outcomes would matter for as well!)
Then comes all the testing - how to test the bot for the content/skills you are trying to support students with (to try and build in the necessary guardrails to keep key outcomes at the center)?
The key thought of what specifically you want students to demonstrate in the chat is so important - centering the specific student learning/outcomes, not just the experience with the bot!
I really appreciate the analysis that goes into the chat history after the interview to really make explicit the critical thinking skills!
The funny thing is -- sometimes I find that attaching background information leads to worse performance - but it is heavily dependent on context.
For example, for Gatsby Bot, I purposely didn't attach any literary analysis of Gatsby's character because it "leads" the bot into conclusions that don't necessarily exist in the book. For these interactions, I'd rather my students be the ones coming into the interaction with hypothesis in hand - and then determining their own conclusions after interacting with a bot that is "less" directed by existing literary analysis.
It depends so much on what you want. But your outline here is exactly how I think about it too!
This makes me wonder if, in the future, classes will have to be completely restructured to allow more time. Right now, so many lessons are rushed because there’s too much content to squeeze into a semester or a year.
In higher education, it’s interesting to think about. AI is speeding up our processes, but could it also push us to slow down and teach more in depth?
The depth of nuance that is available in evaluating these interactions presents an incredible opportunity for expanding metacognition. But it also conflicts with the need to develop and measure foundational knowledge bases and skillsets. Which do we prioritize? Why? How?
In an ideal world, we implement approaches like these once a semester. It's a bit like a final exam -- we "test prep" the students for the chat the same way we "prep" them for an essay, lab report, business case study, or final exam.
Instead, most approaches that I have witnessed seem to focus on "after-the-fact analyses" of AI outputs. All good and well, but I think it misses the point that the outputs are inherently tied to the inputs. Hence, meaningful thought beforehand (via transcript analysis and other methods) preps students to realize that these interactions are not one-way streets.
Do you see actual separate classes in the future in high school, maybe even junior high, that revolve around AI literacy, Mike. Or do you think it will be so baked in that there won't be a need?
I do think colleges are going to eventually have AI literacy courses that students are either going to need to test out of or take if they don't score high enough, just like they have with introductory English classes and math courses.
I tried to launch a cross-disciplinary AI literacy class at my high school 1.5 years ago, but there wasn't enough demand. @terry underwood has launched one at his school and his blog on the subject is fascinating.
Honestly, my vision would be that you still have the core subjects - but around sophomore or junior year this type of cross-disciplinary, student-led AI literacy course is offered. If I were running a high school, it would be a one semester or one-year course that every student had to take -- like an elective.
From what I hear, this is "somewhat" in development at major universities. But between you and me (and this comment thread), I am not sure that they fully understand what this could be.
Yeah Mike, 1.5 years ago you were way too early, and it's unfortunate because if they would've actually listened to you - well you know what that would have meant.
Thanks for letting me know about Terry. I will have to check out his blog. I agree with you that it needs to be a course subject separate from other classes. I think personally there needs to be a required course in college as well - not something you can test out of or opt out of, with different levels (beginner, advanced, etc.).
I have to admit that last line of yours made me really laugh. I worked in higher ed for nearly 20 years. Trust me, they're not prepared for this. It takes so long for decisions to get made and you have committees upon committees upon committees. Plus, you have each curriculum with its own decisions based on what faculty want, so there's going to be all kinds of debating, arguing and territorial pandering.
Hi Mike - Thanks for sharing this interesting example! I'm curious about whether you created some guard rails to prevent ongoing conversations with the bot. Was it only made available to students on the day of the interviews? Did the URL change or was there some other mechanism for that? Other than the transcripts was there any other way to monitor the conversations? Just mindful of some of the recent stories about problematic relationships forming between students and bots and the deterioration of the bot's adherence to guard rails after a certain number of interactions.
Love seeing examples like this, Mike. How did you all go about building the Jay Gatsby bot? How would a teacher think about making one like this for other characters in other books?
Great question!
Building custom bots for these types of activities is more art than science -- and it's probably the subject for an entirely separate post (or workshop series).
But a couple key thoughts:
1. What do you want the student to demonstrate in the chat? Build the bot to behave in a way that creates those opportunities.
For example, for Holden Caulfield, you may have to toggle the depression a bit. I want my students to be tested but I don't want the bot spiraling out of control. For Gatsby - toggle the grandiosity and abstractions. I want to create opportunities for the student to ask incisive questions about personal motivation that will force the bot to "face up" a bit but also not lose its "personality." For Romeo, toggle the wistfulness and self-pity.
This is also highly relevant in building simulations and role-playing scenarios. For instance, I built a simulation for a political science class where students negotiate with fellow legislators and lobbyists. In that case, the question is "how do I want the other stakeholders to engage with the student so that the student has a fair shot at demonstrating their chops?"
Again, this is more art than science. Perhaps it is time to write up a guidebook! But I perceive that educators are not viewing chatbot interactions as potential tests, which is the first step.
Hope that helps!
It is an art! I do think about some of the base skills that teachers would want/need: what information do you need to provide the bot to have as background information? is that available on the web? do you need to include an attachment? what are the fair use rules for more recent publications? What are additional texts or reference documents that would be helpful to include (for which having clarity on the student outcomes would matter for as well!)
Then comes all the testing - how to test the bot for the content/skills you are trying to support students with (to try and build in the necessary guardrails to keep key outcomes at the center)?
The key thought of what specifically you want students to demonstrate in the chat is so important - centering the specific student learning/outcomes, not just the experience with the bot!
I really appreciate the analysis that goes into the chat history after the interview to really make explicit the critical thinking skills!
I completely agree with this comment.
The funny thing is -- sometimes I find that attaching background information leads to worse performance - but it is heavily dependent on context.
For example, for Gatsby Bot, I purposely didn't attach any literary analysis of Gatsby's character because it "leads" the bot into conclusions that don't necessarily exist in the book. For these interactions, I'd rather my students be the ones coming into the interaction with hypothesis in hand - and then determining their own conclusions after interacting with a bot that is "less" directed by existing literary analysis.
It depends so much on what you want. But your outline here is exactly how I think about it too!
This makes me wonder if, in the future, classes will have to be completely restructured to allow more time. Right now, so many lessons are rushed because there’s too much content to squeeze into a semester or a year.
In higher education, it’s interesting to think about. AI is speeding up our processes, but could it also push us to slow down and teach more in depth?
That is my great hope.
The depth of nuance that is available in evaluating these interactions presents an incredible opportunity for expanding metacognition. But it also conflicts with the need to develop and measure foundational knowledge bases and skillsets. Which do we prioritize? Why? How?
In an ideal world, we implement approaches like these once a semester. It's a bit like a final exam -- we "test prep" the students for the chat the same way we "prep" them for an essay, lab report, business case study, or final exam.
Instead, most approaches that I have witnessed seem to focus on "after-the-fact analyses" of AI outputs. All good and well, but I think it misses the point that the outputs are inherently tied to the inputs. Hence, meaningful thought beforehand (via transcript analysis and other methods) preps students to realize that these interactions are not one-way streets.
But as you note, it takes time to slow down.
Great question!
Do you see actual separate classes in the future in high school, maybe even junior high, that revolve around AI literacy, Mike. Or do you think it will be so baked in that there won't be a need?
I do think colleges are going to eventually have AI literacy courses that students are either going to need to test out of or take if they don't score high enough, just like they have with introductory English classes and math courses.
I tried to launch a cross-disciplinary AI literacy class at my high school 1.5 years ago, but there wasn't enough demand. @terry underwood has launched one at his school and his blog on the subject is fascinating.
Honestly, my vision would be that you still have the core subjects - but around sophomore or junior year this type of cross-disciplinary, student-led AI literacy course is offered. If I were running a high school, it would be a one semester or one-year course that every student had to take -- like an elective.
From what I hear, this is "somewhat" in development at major universities. But between you and me (and this comment thread), I am not sure that they fully understand what this could be.
Yeah Mike, 1.5 years ago you were way too early, and it's unfortunate because if they would've actually listened to you - well you know what that would have meant.
Thanks for letting me know about Terry. I will have to check out his blog. I agree with you that it needs to be a course subject separate from other classes. I think personally there needs to be a required course in college as well - not something you can test out of or opt out of, with different levels (beginner, advanced, etc.).
I have to admit that last line of yours made me really laugh. I worked in higher ed for nearly 20 years. Trust me, they're not prepared for this. It takes so long for decisions to get made and you have committees upon committees upon committees. Plus, you have each curriculum with its own decisions based on what faculty want, so there's going to be all kinds of debating, arguing and territorial pandering.
Hi Mike - Thanks for sharing this interesting example! I'm curious about whether you created some guard rails to prevent ongoing conversations with the bot. Was it only made available to students on the day of the interviews? Did the URL change or was there some other mechanism for that? Other than the transcripts was there any other way to monitor the conversations? Just mindful of some of the recent stories about problematic relationships forming between students and bots and the deterioration of the bot's adherence to guard rails after a certain number of interactions.