Amy Homkes-Hayes, Lead Innovation Advocate
Holly Derry, Associate Director of Behavioral Science
Carly Thanhouser, Behavioral Scientist
Molly Maher, Behavioral Scientist
Behind every good learner is good behavior.
We continue to learn that much of the success students and online learners experience comes from choosing the right behaviors, but that doesn’t make choosing easy. That’s why the Office of Academic Innovation is dedicated to integrating behavioral science principles into teaching and learning. Amy Homkes-Hayes, Lead Innovation Advocate, sat down with Holly Derry, Associate Director of Behavioral Science, to explore how she and her team support Academic Innovation’s portfolio of work including educational technology software, MOOCs, and more.
How you would describe behavioral science at Academic Innovation?
We use Behavioral Science to motivate people or spur behavior change. We draw from a collection of strategies and techniques, which have been studied in lab settings or in the field, and apply them to Academic Innovation’s tools. In our work, for example, we focus on how we want learners to behave to be successful in a course. We look through our behavioral science toolbox for ideas on how to keep learners accessing the material every week, completing assignments on time, or engaging on a deeper level in a discussion board. These are just a few examples of where we apply behavioral science in our work at Academic Innovation.
Behavioral science is often used in healthcare settings, but it’s fairly new to edtech. What have you noticed about health behavior change versus behavior change in education?
We’ve been surprised by the similarities. In health and in education, people are working toward goals. In both contexts, some goals are concrete (think: scores) while others are more conceptual (think: feeling better or studying harder). The actions people take to reach their goals might be different, but the ways to motivate people toward these goals are not.
In addition, a lot of the action in healthcare, and education, happens outside of the classroom, or clinical setting. Both patients and learners need self-regulation skills, sustained motivation, and a sense of belonging to continue their health management or learning independent of physicians or faculty.
I hear you talk a lot about how central motivation is to behavioral science. Why?
Anything anyone does is motivated by something. It’s our job to find that something. ECoach, for example, uses both extrinsic and intrinsic motivators to help students succeed in large courses. While tapping into intrinsic motivation may have a longer lasting and more meaningful impact, it’s not always a luxury we can rely on. (How many people are intrinsically motivated to read the textbook?) The extrinsic motivators we have access to (grade feedback, credit, and extra credit) are often better at encouraging students to do something they may not obviously see value in, at first. For example, after students use ECoach’s Exam Playbook, a metacognitive tool that helps them strategize and plan out their use of study resources before and exam, they may become intrinsically motivated to use it again, but we rely on extra credit to encourage them to give it a try.
In the end, we often rely on a blend of intrinsic and extrinsic ways to motivate learners because we know both are valuable in different ways.
How do you account for the differences in intrinsic motivation among learners. In other words, how do you determine what motivates them?
This is where personalization comes in. Each person is different, and when possible, it’s best to offer people choices that will fit with their situation, needs, and desires. This might mean giving people choices on whether they get credit or not, which parts of a course they complete, the time they need to complete it, and so on.
We can also go to the other end of the personalization spectrum and present students with messages written for their specific characteristics. All we need is the data. A few of our projects are powered by the Michigan Tailoring System (MTS), tailoring software developed here at the University of Michigan in 2008. Our projects that use MTS can collect data from people and then deliver a tailored experience to match why they’re taking a course, what grade they want to get, how motivated they are to get that grade, and so on. We can really tailor anything we want, and soon (summer 2018), all of Academic Innovation’s tools can incorporate MTS, and tailoring as well.
Many people assume the tailored messages are computer-generated — they’re not. We’re not using robots to do this. All of the tailored communication and interventions we use are carefully crafted by the behavioral science team.
How do you guide people toward successful behaviors without telling them what to do?
We respect a learner’s right to make their own choices. We see our role as helping people make informed choices. We don’t tell learners what their goals should be, and we don’t tell them what steps they should take. We may say, “here’s what the research says about the best ways of doing something, and here are recommendations you could follow based on that research,” but then it’s up to each learner what they do next. We never say “you must do this or can’t do that.”
We use behavioral science and tailoring to grab people’s attention, to help them see what’s most relevant to them, and to help them make informed choices. We want to share best practices and help people make decisions that are right for them.
What kinds of methods and techniques are you using?
We use methods from many fields, including Public Health, Behavioral Economics, Social Psychology, and User Experience. We rely heavily on Motivational Interviewing (MI), a counseling strategy that has its roots in substance abuse counseling but has expanded into many areas including education. It’s based on the notion that knowledge does not, in and of itself, change behavior. Motivational Interviewing (along with the ability to tailor messages) enables us to find out what motivates people, to build discrepancy between their desired behavior and actual behavior, and to help them more clearly see the path toward behavior change.
We also often use the behavior change recipe from the book Switch, by Chip and Dan Heath. They use the analogy of an elephant (motivation), a rider (logic and reason), and a path (the environment). The goal is to motivate the elephant (because otherwise the rider gets exhausted), direct the rider (so a motivated elephant doesn’t walk in circles), and clear the path (so that the two can get to where they need to go). This is exactly what we’re trying to do here in Academic Innovation by applying behavioral science to our suite of tools, experiences, and opportunities.
Amy Homkes-Hayes, Lead Innovation Advocate
Sharing the Curricular History of U-M
Academic Reporting Tools (ART 2.0) has lived in the Office of Academic Innovation since 2015. During its time with us, faculty champion and ART 2.0 evangelist Dr. August Evrard, Arthur F. Thurnau Professor of Physics and Astronomy, and the ART 2.0 team have iterated on what we show the U-M community. Since March 2016, CourseProfile has enabled students to view course history on things like enrollment in a course by school or college and pre, concurrent, and post course selections. In addition, CourseProfile shows a subset of Student Evaluation of Teaching (SET) data on topics like how much or little students wanted to take the course, or if it increased their interest in the subject. Starting in November 2016 the InstructorInfo deck was added, enabling students to view a subset of SET questions on U-M faculty spanning topics from clarity to creating a respectful classroom environment. In Fall 2017, we released MajorMetrics, which includes a timeline of how many students have graduated with a particular degree (referred to as majors for undergraduate students) in the last 10 years, as well as statistics on joint degrees (co-majors) and minors.
In each iteration of ART 2.0 we revisit our original vision and mission of the tool to promote a deep and shared U-M curricular history with our community, and to aid students in exploring, discovering, and selecting courses. We are enthused to share since fall 2017, more than 21,000 U-M community members have accessed the tool 230,000 times, demonstrating the sustained growth of the ART 2.0 service. As we look to the future of ART 2.0, we are eager to announce new features that align with our mission and respond to student feedback.
New ART 2.0 Iterations
The first exciting new addition to ART 2.0 are grade distributions. Now, when users access CourseProfile, they will see grade distributions for many U-M courses. In fact, of the 500 most searched for courses in ART 2.0, we have grade distribution data on 496. Showing grade distributions in a university-sanctioned tool is something students have asked for since ART 2.0 launched in 2016. Students tell us seeing grades in combination with the perceived workload of the course (something we also show in ART 2.0) helps them make decisions about their course schedule. Of equal importance, showing grades in our service helps “bust myths” about classes – or otherwise helps dispel the notion that “no one gets an A in this course.” We are eager for students to use grade distribution data in addition to the other rich data in ART 2.0 to help inform their class exploration and decision-making.
The second new ART 2.0 feature worth noting is the feedback feature we have added to the tool. Now, users can tell us if they have a positive or negative experience with ART 2.0, as well as leave comments for the ART 2.0 team. We want to hear from our users, and plan on using the feedback we get when examining additional ART 2.0 features.
Finally, we have improved the ART 2.0 search functionality so users can easily toggle between searching for courses, instructors, or majors. Like all of the software we build in the Office of Academic Innovation, we took user feedback on the search experience in ART 2.0 and made improvements to it- making the process of searching less confusing and easier for our users.
Check out ART 2.0 and the new features we’ve added, and feel free to give us some feedback too. We will keep improving the tool in service of the university and its community.
Rebecca Quintana, Learning Experience Designer
“Visual representations can render phenomena, relationships, and ideas visible, allowing patterns to emerge from apparent disorder to become detectable and available to our senses and intellect.” (Hansen, 2000, p. 198)
A well-known challenge for design teams is being able to get a sense of the “big picture” of a design without a mediational tool or aid (Arias et al., 1997). In the field of learning design, diagrammatic or iconic representations of curriculum design can be valuable because they can highlight the relationships among learning activities, to give the viewer a sense of flow and movement (Quintana et al., 2018). Researchers of Massive Open Online Courses (MOOCs) have just recently begun to make initial forays into creating design representations of MOOC curricula. For instance, Daniel Seaton and his colleagues at Harvard have developed methods that enable the creation of iconic representations of course elements (e.g., videos, textual readings) to promote understanding of relationships among course elements (see Seaton, 2016). Seaton’s methods and approach have inspired much of the work I describe in this blog post.
In Part 1 of this two-part series, I described how we use curriculum storyboards during the early stages of the design process to represent emerging ideas about curriculum design and to give project team members a sense of the curriculum sequence—including its rhythm and cadence. In this blog post, I will share our work in the academic research and development space at the Office of Academic Innovation, and describe our experimentation with two representational formats to understand their potential utility for enabling curriculum designers to understand and reflect on course structure once a course has been designed and has “launched.” I will detail our investigation of the following two design representations: (1) course composition diagrams, which are digital and interactive representations of curriculum design, and (2) beaded representations of course structure, which are tangible constructions made with traditional craft materials, such as beads and straws. Both of these representational formats portray an abstraction of course elements and sequences, and fall under the designation of design representations—the codification of a curriculum design that makes it available for review and critique. The goal of our investigations was twofold: (1) to understand the kinds of insights that could be elicited by each format, and (2) the potential value of eliciting insights among design team members at the conclusion of a design process.
Course composition diagrams
Following the methods described by Seaton (2016), we used Plot.ly, a web-based tool for creating infographics, to visualize the course structure of ten recently launched MOOCs on the Coursera platform. Yuanru Tan, a Learning Design and Accessibility Student fellow at Academic Innovation, manually “scraped” data from each page of each course (e.g., video titles, video length, number of “in-video” questions in a video) and from Coursera’s administrative analytics pages (e.g., average quiz scores) as the basis for the course composition diagrams. We used abstract icons to represent the elements of each course (e.g., videos were represented by orange triangles), displaying them in chronological order. Two examples are shown in Figure 1 (course names have been anonymized).
The visualization elements were interactive; as users “hovered” over an element, they could obtain more details about it, such as title (for a video) and average quiz score (for a quiz). See Figure 2.
We invited members of the ten design teams who had worked on these courses, including faculty members, design managers, learning experience designers, course development assistants, course advocates, media specialists, and marketing specialists, to interact with one or more course composition diagrams (i.e., for courses that they worked on) and to complete a short survey that asked them to comment on their experience of interacting with the representation.
Yuanru Tan and I qualitatively analyzed the survey data, following an iterative process outlined by Creswell (2015). Our analysis of the survey data revealed several themes, however in this blog post, I will focus on only one—aspects of the affordances of course composition diagrams. Design team members described how the representations provided a high-level overview of course structure which revealed the proportion of one element type to another. The language in the responses to our survey echoed the principles of visual design, such as balance, variety, repetition, pattern, rhythm, emphasis, and movement. For instance, some comments related to balance with respect to the distribution of one or more element type. One respondent remarked, “Each module was relatively ‘even’ in terms of the number of content types.” Other comments related to balance with respect to the weighting or preponderance of an element type within a specific part of the course: “I see a similar trend of very heavy reading modules in the middle/end of the course.” Our analysis showed that the course composition diagrams allowed design team members to see “what was there,” from an objective point of view, while enabling them to see the relationship of individual elements to the whole course “composition,” suggesting that design team members were able to understand the course structure in a nuanced way.
Beaded representations of course structure
In a second study, we explored an unconventional format—that of beaded representations—to represent the five MOOCs that are part of the School of Education MicroMasters program, offered on the edX platform. Noni Korf, Director of the Learning Design and Media Team, envisioned the idea while at the edX Partners conference in Paris. Noni said, “While in Paris, I found some striped beads that reminded me of icons for text documents, which inspired the first beaded representation.” She was interested in seeing if the differences between the Coursera and edX platforms might be revealed through the beaded representations: “I was interested in finding patterns in MOOCs—were there ways of describing our course experiences that didn’t involve us diving deeply into the content? Could MOOCs be categorized as molecules with different molecular weights? Or pie chart bubbles with differing ratios of content with which to interact? Looking across our portfolio, I wanted to see if patterns would emerge in courses that followed a strictly linear structure (i.e., our MOOCs on Coursera) and in courses that offer more flexibility for learners (i.e., our MOOCs on edX).”
Our learning experience design team developed a method for representing course structure using traditional craft materials (e.g., transparent, translucent, and opaque beads, and colorful drinking straws), stringing them together in chronological order. Each course element corresponded to a “type” of material (e.g., readings were depicted using striped, opaque beads). Because the edX interface uses a nested structure, with course elements organized in sections and subsections, we depicted each subsection as an individual beaded string. Participants could comprehend the hierarchical structure of the course by viewing the first string of beads, followed by the next string beneath it. The courses we depicted were not entirely linear, and where learners could make choices about where to proceed next, so we created a branching structure in the representation by crimping shorter lengths of strings and attaching them to the stem (see Figure 3).
We invited members of the design team that developed the MicroMasters series, including Dr. Don Peurach, the lead designer and course instructor, and Kathryn Gabriele, a graduate student (co-designer) to participate in a focus group session to view and interact with the beaded representations. Our conversation centered around the insights gleaned through viewing the beaded representations. Following a methodology that was similar to the first study, we analyzed transcripts from the discussion. Four uses for the beaded representations were evident from our analysis of the discussion: (1) participants were able to make curricular connections related to course architecture and pedagogical approaches, (2) participants were able to come to a deeper understanding of the learner experience, (3) participants gained insights about the design process, and (4) participants reflected on the method.
As with our prior work with course composition diagrams, the beaded representations provided a “bird’s-eye view” of course structure, allowed participants to make connections among elements, and enabled them to notice aspects such as pattern and variety. We noted the beaded representations acted as “boundary objects,” allowing participants to externalize ideas, facilitate shared understanding, and bridge conceptual gaps (Arias et al., 1997). Our use of traditional craft materials was intended to introduce an element of playfulness and intrigue into our participants’ experience. One participant remarked on how he had a “visceral” reaction to the beaded representations: “What the beads are doing is driving me to think and rethink.”
We intend to deepen our research around design representations, as we continue to refine our methods. Future work will include creating cluster visualizations of courses that are similar along particular dimensions, such as by content type, sequencing, and other types of patterns. We will continue to involve design teams in our work, asking them for their insight toward understanding the utility of various design representations to support understanding and reflection on design at various stages of the design process.
We look forward to receiving feedback on this work at the American Educational Researchers Association (AERA) conference and at the Conference for Human Computer Interaction (CHI). This will help us to advance our work on design representations as part of the work of academic research and development that is happening at Academic Innovation.
Rebecca Quintana, Yuanru Tan, and Noni Korf will present their paper “Visualizing course structure: Using course composition diagrams to reflect on design” at the 2018 Annual Meeting of the American Educational Research Association (AERA) in New York where it has been awarded “Best Paper” in the Online Teaching and Learning Special Interest Group.
Rebecca Quintana and Kathryn Gabriele will present the work of their co-authors Yuanru Tan and Noni Korf at the 2018 CHI conference: “‘It’s just that visceral’: Eliciting design insight using beaded representations of online course structure.”
- Arias, E., Eden, H., & Fisher, G. (1997, August). Enhancing communication, facilitating shared understanding, and creating better artifacts by integrating physical and computational media for design. In Proceedings of the 2nd conference on Designing interactive systems: processes, practices, methods, and techniques (pp. 1-12). ACM.
- Creswell, J. W. (2015). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Boston: Pearson.
- Hansen, Y. M. (2000). Visualization for thinking, planning, and problem solving (pp. 193-220). Cambridge, MA: MIT Press.
- Quintana, R., Tan, Y., & Korf, N. (2018, April). Visualizing course structure: Using course composition diagrams to reflect on design. Paper to be presented at the Annual Meeting of the American Educational Research Association (AERA). April 13-17. New York, New York.
- Seaton, D. (2016, January 29). Exploring Course Structure at HarvardX: A New Year’s Resolution for MOOC Research [blogpost]. Retrieved from https://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year’s-resolution-mooc-research
This article was originally posted on 3/6/2018 on Inside Higher Ed
James DeVaney, Associate Vice Provost for Academic Innovation
Six years ago, inspired by a big idea to democratize higher education, the University of Michigan (U-M) became a founding partner of Coursera. Massive open online courses (MOOCs) were born. While the issuance of MOOC death certificates by skeptics is only rivaled in frequency by those filed by South Park writers for Kenny, MOOCs consistently find ways to survive and indeed thrive in nurturing environments.
MOOCs are far from dead. Rather, they appear to hatch derivatives. Sean Gallagher of Northeastern University’s Center for the Future of Higher Education and Talent Strategy refers to this as “the new ecology of credentials”, a landscape transforming rapidly as we move from the early knowledge economy to the digital, AI, Gig economy. Which leads those of us close to the action to reflect often upon the original big idea for MOOCs. Typically stating a goal to “democratize” is followed by “access to” something. In hindsight, it’s clear we hadn’t fully considered the potential of what we might be democratizing. What, in fact, are we scaling? Is it content and courses? Curriculum and credentials? Communities and college towns?
With today’s announcement, we are now much closer to saying “all of the above”. MOOCs may have initially provided learners an opportunity to simply peer into the university. Now MOOCs and MOOC derivatives (e.g. Teach-Outs, specializations, MicroMasters, MasterTrack, etc.) are helping universities to expand how they think about engaging with the world. For U-M, this is entirely consistent with top institutional priorities around academic innovation, diversity, equity, and inclusion, and public engagement. We are the global, inclusive, public research university.
The real innovation of the MOOC era is not the unbundling of academic degrees that first captured massive attention, but rather the re-bundling that results from serious academic R&D – the creation of new communities and credentials for all levels. In announcing Michigan’s new degrees this morning at the Coursera Partners Conference, Coursera CEO Jeff Maggioncalda contextualized these latest innovations as evidence that, “the future of work and the future of learning are converging.”
Today U-M announced the intent to design two new fully online master’s degree programs and a new online cohort-based pathway to advanced degrees and career advancement called the MasterTrack Certificate. Let’s consider this latest re-bundling effort within the broader context.
U-M seeks to address global problems in pursuit of a more equitable world. If we can agree that global problems do not fall neatly into the academic disciplines, it should follow that the increasingly diverse needs of learners would be difficult to address through a set of unmalleable academic offerings. If we are serious about diversity, we need to be equally serious about inclusivity as we design new programs, and laying a foundation for learners with vastly different starting points, learning styles, and learning objectives.
So in 2012 we began to adopt ‘unbundling’ as part of our language. Many chose the fear narrative and heard unraveling. We chose the opportunity narrative and have been re-bundling ever since with an evolving mix of learner-centric offerings. Because experiments regularly fork into new experiments, it is easy to lose focus. As we move at a rapid speed, we find it is critical to anchor in our vision for a preferred future – one that points U-M in the direction of expanding access, designing for inclusivity, personalizing at scale, and reimagining two-way public engagement. We took a major stride toward this future today by announcing two new degrees and pioneering the MasterTrack offering. Along with our MOOC portfolio, our expanding Teach-Out Series, and our MicroMasters programs, learners have more opportunities than ever to be a part of a Michigan learning community.
We’re just getting started. And the world future of work and future of learning show no signs of slowing down. Given what is known and all that is uncertain, our goal is to build a global, inclusive, public research university that is future proof!
As we move toward this future, it’s clear that there is a time for acceleration and a time to struggle through experimentation – advancing learning and recording failure along the way as only Universities do.
And now, an adorable tangent on barriers to entry, speed and pace.
My six-year-old daughter is beginning to love soccer. Before each practice, I ask her how she will train today. She pauses predictably, smirks and tilts her head to the right, and with one eye visible responds, “like a cheetah-rocket!” Never heard of one? Well, for those who haven’t spent time with a six-year-old lately, these are two things that are, like, really fast. So when you put those two things together, it would stand, that you get something even faster.
When my daughter first showed signs of doubt that she could compete with the “big kids” (seven and eight-year olds are Goliath to a six year old David), we focused on getting in the game. She needed to belong. We talked about the way she would enter the pitch. After some epic brainstorming, she refused to choose between ‘like a cheetah’, and ‘like a rocket’. “A cheetah-rocket would be faster, Dada,” she stated decisively. She’s now in the game. She wants to try and she know she can. She’s ready to learn.
I think about our learning curve on the soccer pitch often as the higher education industry evolves with competing narratives of opportunity and fear.
At the moment, I’m helping my daughter to gain confidence. Soon I’ll need to help her understand that learning is hard and that part of the human experience is to struggle through new lessons. We’ll need to slow the game down to understand each component. Speed and learning don’t often go well together.
We are steadily lowering the barriers to entry in higher education. We unbundle to grant cheetah-rocket speed to all. Access, belonging, opportunity, personalization. As we re-bundle, we need to create new opportunities that advance learning rather than enable the tyranny of convenience. This will take serious experimentation in order to establish the best mix of learning opportunities and credentials for the economy ahead.
With my daughter, I need to help her slow down and understand the fundamentals at her own pace in order to lay a sound foundation for learning. Similarly, MOOCs provide a foundation for self-paced learning. As we continue to experiment, we need to make sure this foundation is flexible.
Importantly, for universities in this moment, it turns out that speed as a lever goes in multiple directions. Universities need to continue to gain comfort with good risk taking. A burst of cheetah-rocket speed now and then can help us to accelerate experimentation in pursuit of our ultimate goals. Yet we also need to apply good methods and R&D principles to make sure we pace ourselves when appropriate and ultimately reach our desired destinations. Do we have the confidence to set the right pace and embed good pedagogy as we continue extend our reach? We have made significant progress in expanding our reach, but we haven’t yet cracked the code on embedding good pedagogy at scale. This will be a primary focus in the next wave of experiments.
For U-M, we’ve envisioned a preferred future that allows us to be more global, more inclusive, more public. Many ways in. Several ways through. Clear outcomes and value.
Today we took another step forward in the great re-bundling and it is clear that there is a long road ahead. Good things take time. As we continue to experiment and design learner-centric programs and learning communities, we intend to make design choices that support informed decision-making for learners, increase affordability, increase acceleration, increase frequency and quality of feedback, and replace a capstone mindset with project-rich learning experiences throughout.
Experimentation is far from over. As we launch this latest set of programs, several questions are on my mind:
- Given what we expect in the future of work, can we create pathways to continuous competency?
- How will our evolving product mix fit together for different kinds learners?
- Is there tension between access/on-ramps and deep learning?
- How should we incorporate real-world projects into rich, rigorous, and agile curricula?
- How will employers evaluate sub-degree credentials?
- What are the best ways to engage learners and employers in the design of learning experiences?
- Are we addressing the audiences that need us most?
So what are we democratizing? It turns out for U-M, our efforts are focused on scaling the great public research university in pursuit of a more equitable world. Neither MOOCs nor degrees are dead. Instead, we have entered an era of experimentation that will result in a new collection of credentials needed in a future where, as Mark Searle, Arizona State University Provost said so memorably in his keynote this morning, “universities are known for who we include not who we exclude”.