'Grade The Chats' is the 'Show Your Work' of the AI Age
A Process-Based Approach to Assessing Student Thinking and AI Literacy
***This post was written with the assistance of ChatGPT.***
I wanted to take a moment to thank you for your continued support AI EduPathways!!! Please consider becoming a paid subscriber.
Take advantage of the yearly rate discount. $50 at year = $4.16 per month.
Five months ago, the concept of “Grading The Chats” was introduced as a way to assess student learning in an era where AI tools can produce high-quality outputs. Since then, educators at all levels have experimented with this approach to tackle a pressing challenge: how to evaluate what students truly know and understand when AI can do much of the heavy lifting.
Now, a new layer of understanding has emerged: “Grading The Chats” can serve as the “Show Your Work” of the AI era.
Much like how math teachers responded to the advent of calculators, this method shifts the focus from simply evaluating the final product to analyzing the thought process behind it. When calculators first became widespread, many feared they would erode students’ ability to think critically about math. The solution was to implement “show your work” policies, where students had to demonstrate their reasoning and problem-solving steps. This allowed teachers to assess the process, ensuring students understood the underlying concepts even as they used calculators for computation.
Similarly, in today’s classrooms, AI tools can assist with tasks ranging from essay writing to problem-solving. By evaluating chat transcripts between students and AI, educators can observe how students interact with these tools, making their thinking visible in a way that wasn’t possible before. This approach not only allows for a deeper understanding of students’ skills but also fosters the development of critical AI literacy.
The Value of Process-Based Assessment
When we evaluate the process rather than the final product, we gain valuable insights into how students think, question, and solve problems. In the context of AI, this means looking at:
Prompt Writing: What kinds of questions or tasks are students asking the AI? How well do they communicate their desires/needs? How well do they contextualize their prompt?
Breaking Down Problems: How do your students approach problems? Do they break them down into sub-components before or during the chat? Can we build metacognition and reflection through this evaluation?
Response Analysis: How do students evaluate the AI’s responses? Are they accepting answers at face value or critically analyzing them?
Iteration and Refinement: Do students “think on their feet?” Do they build from, pivot, or redirect the AI system towards more accurate and valuable outputs? Do they modify their own thinking?
This kind of process-based assessment aligns with established educational practices. In mathematics, for example, word problems are designed to test a student’s ability to apply concepts in complex, real-world scenarios. Similarly, in AI-assisted assignments, the task should be broad and open-ended enough to require deep engagement.
Practical Strategies for Implementing “Grading the Chats”
Educators who have experimented with this methodology offer several practical insights:
Design Big, Ambiguous Tasks
Just as math teachers create complex word problems, educators can design tasks that require students to engage deeply with AI. These could be open-ended research projects, complex case studies, or creative writing assignments where the scope and ambiguity demand critical thinking and thoughtful AI use. “Small” tasks that AI can complete easily often prove ineffective within this framework.Provide Clear Expectations
To help students understand what is expected, it’s crucial to provide examples of effective AI interactions. Consider creating fictional chat transcripts that illustrate strong prompt-writing, thoughtful responses, and critical engagement. A model class activity for this exercise is available for free on the home page of our website. Some educators even share their own chats and ask students to “grade” them using a rubric. This not only sets clear expectations but also models the reflective process.Offer Feedback on the Process
Feedback is essential for learning, and students are often eager for guidance on how to improve their AI interactions. Providing constructive feedback on their chat transcripts helps students understand what they’re doing well and where they can improve. This mirrors the way math teachers provide feedback on the steps of a solution, not just the final answer.Start Small and Scale Up
You don’t need to incorporate this method into every assignment. Start by requiring students to submit chat transcripts as part of a larger project without assigning a grade to the chats themselves. This allows both you and your students to get comfortable with the process. Over time, you can integrate “Grading The Chats” into your assessment framework more fully, perhaps using it for one or two major assignments per semester.Reflect on Your Own Practice
Before implementing this approach, try grading your own AI interactions. Pull up a chat from a past project and analyze the user on the page as if it were a student. What do you notice about your prompt-writing and response evaluation? This exercise can help you develop a stronger understanding of what thoughtful AI use looks like and how to guide your students effectively.
Learning from the Field
In my next post, I will share anecdotes from educators in the field who have experimented with this methodology and found success. They are included but not limited to; Jason Gulya, Michelle Kassorla, Doan Winkel, Bruce Clark, Aaron Makelky, and Patricia Vining.
Have you graded the chats? Have you asked students to submit their chats? Do you have questions about how to implement this methodology? Reach out!
Conclusion
As AI becomes an integral part of education, it’s essential to adapt our assessment practices to keep pace. “Grading The Chats” offers a way to do just that, allowing educators to focus on the process of learning rather than just the final product. By drawing parallels to the successful integration of calculators in math education, we can see how process-based assessment can help students develop the skills they need to thrive in an AI-driven future.
For those ready to take the plunge, start small, experiment, and see where it takes you. The insights you gain will not only benefit your students but also deepen your own understanding of how to teach and learn with AI.
Evolve Your School’s AI Landscape
Zainetek Educational Advisors provides consulting, curriculum design, and professional development services to educational institutions and organizations seeking to upgrade their approach to AI adaptation.
Book a free consultation on our home page to discuss pathways and solutions for your school or organization.
Zainetek’s School Adaptation Framework:
Reach out for booking information today: mike@zainetek-edu.com
Grade the Chats should be a book title.
This is definitely an approach that makes sense and meets students where they are. Getting there, though, takes time. As I've managed teachers, I immediately think of several who would need a structured understanding for their own selves and then a way to bring that to students. It's a different skill to analyze chats than to analyze writing, and for some that difference is vast. PD that walks teachers through that process is integral to everyone—students, teachers, admin—being successful.