The Assessment Toolkit Initiative
How do we know that our programs, initiatives, degree programs and academic innovations help students?
This question is central to a broad range of initiatives in higher education and is an important pillar of our efforts to create equitable and inclusive spaces for student success. Yet, this question is remarkably difficult to address for many programs, units, and investigators. Access to the institutional datasets takes effort; the data itself is challenging to understand; even relatively simple-to-ask questions ‘How do students participating in X do, compared to otherwise equivalent students not involved in X?’ take real expertise to frame and code. Where these conditions exist, data has been able to illuminate important challenges and successes (e.g., Foundational Course Initiative’s equity analyses, or M-ENGIN’s assessment). Yet, we lack the capacity to assess many of our efforts in principled ways, making it difficult to understand where we are succeeding and where we could benefit from changing tack.
Our vision for the Assessment Toolkit Initiative is to reduce the challenges around engaging in this work. We aim to do this through:
- Creating rich documentation on existing datasets and access practices
- Sharing worked examples, notebooks, and sample code to document common analytical practices
- Developing a community of people engaged in this work, creating opportunities to discuss goals and approaches with peers, and learn from their experiences. Community members will ideally include technically-minded people, program directors/PIs, researchers, and administrators from across the University.
What does success look like? In a few years, one could hope that:
- Instructors and departments want to understand what happens to their courses – They are able to perform an equity analysis and take part in a group that is concerned about the same issues and can help them understand and interpret what they learn.
- Directors or PIs of our initiatives can use the right computational and statistical tools to document the impact of their efforts – who is thriving, and who needs more attention? Did an initiative help anyone, compared to an otherwise similar group who was not involved?
- Discipline-based education researchers have a platform that dramatically lowers the entry cost towards using our rich institutional data; this helps to shape and generate the next generations of questions and insights that help us further refine our support of student learning and success.
Every student at the University of Michigan deserves to have the opportunity to be successful, regardless of their background. Yet, persistent inequities in higher education and society prevent that from happening. Our goal is to identify inequities in courses, departments, and degree programs in order to eliminate them.
To do so, we are developing a set of utilities that allow us to use university records to analyze and explore outcome inequities across campus. We aim to share these utilities with the broader community, to enable stakeholders across the university to engage in this reflection and data-informed iteration. It is only with thorough, transparent, and thoughtful evaluation that we will succeed at both identifying inequities and taking the necessary steps to address them. We call the collection of tools developed for this purpose the Assessment Toolkit.
What’s currently in the toolkit?
Course Equity Reports
First created by Heather Rypkema (Head of Learning Analytics at CRLT) to help courses embarking on the Foundational Course Initiative program of transformation, Course Equity Reports leverage five years of enrollment and outcome data in a given course to reveal patterns in student engagement. The Assessment Toolkit team has made it possible to produce these data-rich reports easily for any given course on campus. With these data in hand–and importantly, explorable through adept usage of visualizations–we can then highlight potential curricular changes that might be of use to address observed inequities. See a sample report.
Major/Department Equity Reports
While some inequities can be observed within individual courses, others are apparent when we go up a level and consider student experiences across a complete program. Building on the utility of the Course Equity Reports, we are currently working with stakeholders to design Major/Department Equity Reports to consider student outcomes at this higher level. Key metrics include attrition across the major and GPA both in and out of the major. See a sample report.
Our Team
- Eric Bell (LSA)
- Carson Byrd (SOE)
- Susan Cheng (COE)
- Nate Cradit (CAI)
- Holly Derry (CAI)
- Cait Hayward (CAI)
- Ben Koester (LSA)
- Becky Matz (CAI)
- Mark Mils (CAI)
- Heather Rypkema (CRLT)
- Nick Young (CAI)
Connect with us
Have a question or concern? Reach out to us at [email protected]!