Redesigning Literary Assessment for the AI Age

How Character Chatbots Became Critical Thinking Tools

As educators, we're living through a moment that demands both humility and experimentation. The rapid emergence of generative AI has left many of us scrambling to understand how these tools might transform our classrooms—and whether that transformation will ultimately serve or undermine student learning.

Rather than waiting for definitive answers, I've been working to test new approaches based on my years in the high school English classroom. What I'm sharing today isn't a polished solution, but rather the results of one experiment that showed promise: a five-day lesson sequence where students interview a character chatbot — in this case, “Jay Gatsby” — as a means of deepening their character analysis skills.

An overview of this approach was originally published in The Harvard AI Pedagogy Project, but I'm now offering the full implementation details for educators who want to test similar approaches in their own contexts.

Download the full plan in the “English Language Arts” section of this page.

This is part of a new offering of assessment redesign services I'm developing in partnership with three current classroom teachers who bring varied expertise across four major disciplines. Wess Trabelsi, Ned Courtemanche, and Matt Karabinos are joining me in sharing their classroom-tested plans to support educators in their efforts to navigate around AI cheating, test AI-based instructional materials, and discern the best way to conceive of the educational experience in the age of AI.

The Challenge We're Facing

The traditional essay-based character analysis assignment has become vulnerable to AI shortcuts. Students can now generate sophisticated literary analysis in seconds, making it difficult to assess their actual understanding of complex characters like Gatsby. But instead of banning AI or returning to handwritten assessments, I wondered: what if we could redesign the task entirely?

The key insight came from recognizing that while AI can write about characters, it struggles with the nuanced, real-time thinking required for substantive interviews. This led me to experiment with having students become literary journalists, conducting live interviews with character chatbots and then critically analyzing the experience.

How the Experiment Works

The five-day sequence centers on students preparing for, conducting, and reflecting on an interview with a Jay Gatsby bot. Here's the basic structure:

Days 1-2 focus on preparation. Students craft open-ended interview questions designed to probe Gatsby's motivations and contradictions—the "unanswered questions" of the novel. They practice conversational interviewing techniques: building on responses, pivoting when needed, and pressing when they detect evasions or inconsistencies.

Day 3 is the formal interview. Students engage in 10-14 conversational turns with the bot, treating it as a live interview subject. The goal isn't to stump the AI, but to demonstrate deep character understanding through thoughtful questioning and follow-up.

Days 4-5 involve critical analysis. Students annotate their transcripts using a provided legend that helps them identify moments where the bot deflected, contradicted itself, or revealed limitations. They conclude with a reflective essay comparing the bot's responses to their understanding of Gatsby from the novel.

What Made This Work

Several design principles proved crucial to the experiment's success:

The bot as subject, not tutor. Rather than positioning AI as a learning aid, students approach it as an interview subject with potential blind spots and inconsistencies. This immediately puts students in the analytical driver's seat.

Emphasis on the unanswered. The best questions push into territory the novel leaves ambiguous—Gatsby's missing years, his deepest motivations, his capacity for self-reflection. These gaps force both student and bot into uncharted territory where critical thinking becomes essential.

Multiple assessment layers. The transcript alone doesn't tell the full story. Combined with question preparation, post-chat annotations, and metacognitive reflection, the assessment creates a comprehensive picture of student understanding that would be difficult to fake or outsource.

Built-in AI literacy. Students don't just use AI; they critically evaluate its limitations. The reflection essay often becomes a sophisticated analysis of how the bot both succeeded and failed to capture Gatsby's complexity.

Early Results and Observations

In my classroom implementation, student engagement was notably high. The real-time nature of the interview created genuine intellectual pressure—students couldn't simply retrieve pre-written analysis but had to think on their feet. Many reported that the experience helped them discover gaps in their character understanding that traditional essay prompts hadn't revealed.

The transcripts themselves became rich texts for analysis. Students identified moments where the bot deflected difficult questions, contradicted earlier statements, or revealed algorithmic limitations. These observations often led to deeper insights about the character and the nature of unreliable narration.

Perhaps most importantly, the assessment proved resistant to AI shortcuts. While students could theoretically use AI to help generate initial questions, the live interview format and requirement for authentic response to bot answers made the core intellectual work difficult to outsource.

The Research Context

I'm sharing this experiment as part of a broader effort to work with educators who want to test similar approaches based on the principle of "AI Interactions as Tests." The University of Washington's Master's in Communication Leadership program represents one early partnership in this work, though we'll be exploring different applications there. This Gatsby interview represents one data point in a larger investigation of "amplify, not accelerate" approaches to educational technology.

The lesson plan materials I've developed—including detailed rubrics, annotation legends, sample questions, and implementation guides—are available for other educators who want to test this approach in their own contexts. But I want to be clear: this is offered in the spirit of collaborative experimentation, not as a definitive solution.

Looking Forward

What excites me most about this experiment isn't that it "solves" the AI challenge in education, but that it suggests a different way of thinking about the problem. Instead of seeing AI as either threat or savior, we might approach it as a new kind of text to be critically examined. One that can reveal as much about student thinking as traditional assessments, perhaps more.

I'm eager to partner with other forward-thinking educators who want to test variations of this approach. How might similar interview formats work with historical figures in social studies classes? Could science students interview "expert systems" about complex phenomena? What would happen if math students had to question AI about problem-solving strategies?

The goal isn't to find the perfect AI-integrated lesson, but to build a community of practice around thoughtful experimentation. If you're interested in testing this approach—or sharing your own experiments with AI-enhanced assessment—I'd welcome the conversation.

Sometimes the best way forward is to wade in carefully, document what we learn, and share our findings with humility. That's the spirit in which I offer this lesson plan: not as the answer, but as one educator's attempt to navigate uncharted territory with students' critical thinking at the center.

---

The complete lesson plan materials, including rubrics, handouts, and bot setup instructions, are available through AI Literacy Partners' research consulting and assessment redesign services.