Exam Playbook Improves Student Performance in Real-World Course Settings

New research highlights the effectiveness of social-psychological intervention and takes it from controlled trials and into live courses

Sean Corp, Content Strategist

While there is ample research on the efficacy of social-psychological interventions to boost student performance using controlled trials, new research shows that those interventions had a substantial effect across diverse STEM courses in real-world settings. 

The research, published in the Nature Portfolio Journal npj Science of Learning, utilized the University of Michigan Center for Academic Innovation software ECoach and its Exam Playbook tool to help high-stakes exam preparation, with students using the tool showing a 2.17-point boost in scores. 

ECoach is a personalized, web-based platform that provides targeted interventions to students at scale. The Exam Playbook within ECoach allows students to access course-specific resource checklists and prompts them to think through how each resource helps their studies. This self-reflection helps students map out a personalized study plan. 

Researchers made the Exam Playbook learning resource available to more than 12,000 college students in 14 introductory STEM courses across two semesters. Results showed a 2.17-point increase on average in exam scores for those who utilized the Exam Playbook, and the impact was positive for 13 of the 14 classes examined. Subsequent analyses also showed that using the Exam Playbook reduced the STEM gender achievement gap by over 60% and the first-generation achievement gap by 30%, potentially offering a promising way to reduce disparities.

Cait Hayward
Cait Hayward

“These results emphasize how adaptable and effective the Exam Playbook intervention is. It’s easy to implement and deploy with a tool like ECoach, and has a reliably positive effect across all of these contexts despite it taking students just a few minutes to complete,” said Cait Hayward, one of the authors of the study and the director of research and analytics at the center.

Translating research into a tool to help thousands of U-M students

ECoach supports large courses where one-on-one communication between instructors and students is otherwise impossible. Tim McKay, Arthur F. Thurnau Professor and associate dean of undergraduate education in LSA at U-M, and a team of researchers developed the tool in collaboration with the Center for Academic Innovation.

The new paper builds on previous research first conducted in 2013 when Patricia Chen, then a doctoral student in psychology at U-M, now an assistant professor in the Department of Psychology at the National University of Singapore, showed that a simple social-psychological intervention could provide as much as a one-third boost in letter grades. Her findings were published in Psychological Science in 2017, and with this research in hand, the center built the Exam Playbook within ECoach. 

“At first glance, the core of the Strategic Resource Use intervention seems to be straightforward—giving students a list of the learning resources available in the class and asking them which they would choose and why. It precisely unlocks a key psychology behind self-regulated learning,” said Dr. Chen. “By articulating why they would use a specific resource, students become more purposeful about their studying. They do not necessarily study longer, but each hour of studying becomes more effective, resulting in lower test anxiety and, eventually, better exam scores.

Chen is the lead author of the new research paper, with co-authors Dennis W.H. Teo, Daniel X.Y. Foo, National University of Singapore; Holly Derry, Ben Hayward and Cait Hayward of U-M’s Center for Academic Innovation; McKay of U-M; and Desmond C. Ong, of The University of Texas at Austin.

Testing Effectiveness in a Real-World Setting

One of the advantages of building the Exam Playbook into ECoach was the ability to take that Strategic Resource Use intervention, which was previously tested in two randomized controlled trials, and test its effectiveness in a real-world setting. It also created an opportunity to see how effective these interventions were within a student population at an age where learning becomes increasingly self-managed. 

“The intervention had a positive impact in its first test–Introduction Statistics–but the center’s partnerships across campus give us an incredible opportunity to research how well this works in multiple contexts. Thanks to Dr. Chen and her team, we can now show in detail how effective this intervention is, in which contexts, and for whom,” said Hayward. “We’re excited to push this further, including studying the effectiveness of the Exam Playbook at other institutions and testing additional interventions that could support student success.”

Showing real-world applicability of the interventions and scaling it up for more significant impact is the power behind using educational technology within educational research, said James DeVaney, founding executive director of the Center for Academic Innovation. 

James DeVaney
James DeVaney

“We’re proud to show that Dr. Chen’s idea for an intervention, turned into a feature in ECoach, benefited thousands of University of Michigan students, and congratulate the authors on their paper,” DeVaney said. “This collaboration allowed us to tackle a real-world problem identified by faculty, create research- and theory-based interventions to address them, and build cutting-edge technology to support the students who need it most at the right time.”

Engage with the Center for Academic Innovation 

Faculty and staff interested in using educational technology to advance teaching are welcome to explore the tools available at https://ai.umich.edu/educational-technology or to connect with the center to receive more information or demonstrations by emailing AISoftware@umich.edu.

Recent Posts

Learn the Art of Giving and Receiving High-Quality Feedback

Read More

Learn the Health and Environmental Impacts of the Foods We Eat and Products We Buy

Read More

Higher Ed, Industry Leaders Join the XR at Michigan Summit to Discuss the Future of XR in Education

Read More