Empowering Educators: Harvard Professor on AI Tools as Effective Tutors

Two speakers sit onstage at an Innovation Event. A man in a light blue shirt gestures while speaking, and a woman with long blonde hair holds a microphone and smiles. Behind them, a screen displays the text, "Tune the AI-tutor based on prompt engineering experiments." — Chris Brooks, left, associate professor at the University of Michigan School of Information, fielded questions for Harvard University’s Kelly Miller following her AI Futures keynote presentation at the Center for Academic Innovation. Photo by Charlyn David | Center for Academic Innovation

AI Futures keynote series kicks off with Harvard’s Kelly Miller sharing her questions — and answers — about the benefits of AI tutors

Mindy Arbaugh, Content Specialist

As Kelly Miller neared the end of her presentation, the first installment of the AI Futures keynote series at the University of Michigan’s Center for Academic Innovation, she returned to one of her experiments that tested an artificial intelligence (AI) tutor’s impact on students. In explaining the results — a whopping 62% increase in learning gains — Miller went back to where she started. She asked a question.

“What are the things that the platform is doing that could lead to these results?” said Miller, a faculty member at Harvard University’s John A. Paulson School of Engineering and Applied Sciences. “I’ve thought a lot about why there was such a difference.”

That cycle of asking questions, leading to more questions, is why Miller was a fitting speaker to kick off the center’s AI Futures keynote series, featuring conversations with higher education leaders about how AI is reshaping teaching and learning.

“Kelly brings together classroom experience, rigorous research into what students learn, and a willingness to test new possibilities,” said James DeVaney, the center’s founding executive director and associate vice provost for academic innovation at U-M. “We are thrilled she is sharing her expertise with our Michigan community for open discussion and debate as part of CAI’s AI Futures keynote series.”

Miller walked the standing-room-only audience through her research on AI tutors, including recent work with Collage AI, a customizable AI teaching and learning platform for which Miller serves as chief learning officer. During her presentation, “Designing AI-Enhanced Learning: Evidence, Pedagogy, and Student Use,” Miller detailed how AI can not only adequately support students, but do so with surprising efficacy.

Kelly Miller, a faculty member at Harvard University’s John A. Paulson School of Engineering and Applied Sciences, presents at the center’s first edition of the AI Futures keynote series. Photo by Charlyn David | Center for Academic Innovation

Researching AI Tutors

Miller’s questions first led to targeted experiments on AI tutors. She and her colleagues in Harvard’s physics department randomly divided students in a large introductory class, instructing half to attend in-person classes normally, and assigning the other half to stay home and learn from a custom AI tutor.

The team then flipped the students for the subsequent lesson and reviewed their pre- and post-test results. Their findings indicated that average learning gains were nearly double for those using the AI tutor compared to in-person classmates. The results gained national attention, and Miller and her colleagues took notice.

“This was a very well-taught class that had been iterated on over many years,” Miller said. “It represented the best pedagogy that the Harvard physics department had to offer, and so this was surprising to them and to us.”

Miller’s additional research on the AI tutor revealed that tools built with enhanced prompts — guiding the tools to use pedagogical best practices such as active learning and cognitive load management — had more student interactions.

“We set out to measure the efficacy of AI tutors for promoting learning and engagements, and it forms the early groundwork for some of what we’ve built into Collage AI,” she said.

Knowing what was proving effective, Miller turned to her next question: could Collage AI do a better job preparing students for in-class work?

A man in a navy shirt holds a microphone and speaks from his seat during an audience discussion. Attendees seated around him listen attentively in a modern event space with glass walls and exposed brick. — An audience member asks a question of Harvard University’s Kelly Miller following her packed-house AI Futures keynote presentation. Photo by Charlyn David | Center for Academic Innovation

A Better Flipped Classroom

Miller integrated Collage AI into one of her applied physics courses, which is designed as a flipped classroom, meaning students learn concepts through pre-class materials and then use the classroom time for practice and collaborative learning.

She fed the content she typically assigned for the pre-course work along with the learning objectives into Collage AI, then used it to create formative assessments and exercises. The platform also served as an AI tutor for students, guiding them through their questions.

The results were positive. When surveyed, students appreciated the feedback they received from the AI tutor. More striking were the post-test gains: compared to a previous semester, students using the Collage AI tutor had 62% higher learning gains.

Spreading the Benefits

Miller said one of her favorite Collage AI features is the level of control faculty have over the tool. Instructors have multiple points during development to decide what the lesson looks like, what kinds of assessments to include, which level of Bloom’s taxonomy to target, and so forth.

Miller is now taking her questions to other instructors. She is currently working on implementing Collage AI into different courses outside the physics department, as well as at other universities and in K-12 classrooms.

“We’re continuing to gather information that will allow us to tune the AI tutor and the platform in general toward promoting learning outcomes for students,” Miller said.

AI Futures Keynote Series Continues

Building AI Pedagogy with Socratic Tutors

Keynote: Julie Schell, University of Texas at Austin

When: Nov. 17, 2026, from 1-2 p.m.

Where: Rogel Ballroom at the Michigan Union

Developing Relational Intelligence in the Age of AI

Keynote: Isabelle Hau, Stanford University

When: Jan. 27, 2027, from 1-2 p.m.

Where: Pendleton at the Michigan Union