Jeremy Nelson, Director of XR Initiative

In this week’s MiXR Studios podcast, we talk with Joanna Millunchick, the Arthur F. Thurnau Professor of Materials Science and Engineering and the associate dean for undergraduate education at the University of Michigan’s College of Engineering. Joanna teaches courses in the Department of Materials Science and Engineering and explores crystal structures in virtual reality. She is also the associate dean of undergraduate education and is interested in understanding how we can advance undergraduate curriculum to include XR technologies to support teaching and learning.

Her work in XR was inspired by the initial explosion in popularity of Pokemon Go back in 2016,  about how augmented reality could be used in education to help transform student learning. That led her to explore what software and technologies could be adapted for learning, and ultimately to an Ann Arbor company called Gwydion. Gwydion, founded by former U-M students, built software to support educators in teaching and researching 3D space with XR. Unfortunately, Gwydion went out of business, but the university acquired the rights to its application Arthea and hired one of the key developers, Talal Alothman, to support and extend that platform.  

We discuss her thoughts on how the normative context for XR technologies is still not well understood and where there is still a lot of work to be done. Other technologies such as mobile phones, websites, and computers have a well known normative context thanks to decades of use and often don’t require training users. She has been exploring how VR impacts the way students establish spatial reasoning skills and retain knowledge. In her early findings, VR is helping students understand how structures are constructed but the long term retention of that knowledge doesn’t persist. She continues to explore how best to deploy this type of technology and where in a student’s learning journey does it make sense to bring in a tool like this. 

As the associate dean of undergraduate education, Joanna created an XR clinic within the College of Engineering to support faculty in experimentation. This clinic allows faculty to consult with Talal to understand their research and teaching to evaluate how best to deploy and develop XR technologies. It is important to research how these technologies help students learn, so Carolyn Giroux, a PhD student from the School of Education, performs research on how these projects are affecting learning. The College of Engineering has funded XR projects ranging from visualizing planning for outreach in a natural disaster to performing customer discovery tools with 360 video in VR with voice recognition and AI.

Our talk continued into the challenges around content creation and developing a normative context on how students and faculty can create capacity. Joanna shares how Michigan can lead the way in understanding how XR can improve learning outcomes across multiple domains. We discuss how COVID has accelerated the timeline for embracing XR technologies and how the challenge will be to get students access to the devices to continue to design and develop new experiences. Now is the time for Michigan to lean further into this space and push higher education forward on a more immersive and collaborative path.

I learned so much in my conversation with associate dean Millunchick about how she is researching XR in her teaching and how to apply that more broadly across the university. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 18

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson. And today we are talking with Joanna Millunchick, who is the Arthur F. Thurnau Professor of Material Science and Engineering, and the associate Dean of Undergraduate Education for the University of Michigan’s College of Engineering. We are talking about bringing VR to her crystal structures course and how she is thinking about XR more broadly to advance undergraduate education coming up next in our MiXR podcast.

Jeremy Nelson (00:31):
Hello Joanna, thank you for joining us.

Joanna Millunchick (00:44):
It’s my pleasure. Thanks for having me.

Jeremy Nelson (00:46):
Yeah. I’m excited to talk with you today and share your story and the work you’ve been doing with our audience. And just to kind of learn more about what you found out as you’ve been teaching with XR and kind of how you’re thinking about things more broadly, would you mind sharing, like how you got into XR? Like where did your journey start?

Joanna Millunchick (01:08):
Oh, you know, it’s actually hilarious because I didn’t, I was teaching in material science for a long time and material science, for those of you who don’t know what it is, it’s really the, the science of stuff, you know, how do the different materials, everything from metals to ceramics, polymers, natural materials, like wood and so forth, how do they get their properties? And the answer is that their properties come from their structure and structure is a three dimensional thing. And so I’ve been struggling for a long time to help students understand what these three dimensional structures actually look like. And you know, the concepts aren’t difficult. What is difficult is to convey exactly what you mean in 3D space, because we’re used to thinking about like things on a page or, you know, things written down as equations. And so, you know, getting that kind of physical intuition, I found to be really difficult.

Joanna Millunchick (02:08):
So a couple of years ago you remember Pokemon Go, right?

Jeremy Nelson (02:12):
Oh yeah. Oh yeah.

Joanna Millunchick (02:13):
So Pokemon Go kind of opened my eyes to, Oh my gosh. You know, like, augmented reality, maybe we could, we could do it that way. Because so many people were using it on their phones and, you know, it had like this three dimensional nature to it and it’s situated these characters in like real space, if you will. And I was thinking, you know, what, if I were to put a crystal structure like in front of the students like that. And if, if I were to like place them in front of them and they can work on, you know, looking at the structure and doing their homework at the same time, that would be incredibly powerful. So that’s what gave me the initial idea of trying to go into it. I am not a gamer by any stretch of the imagination, you know, so, you know, it’s not like, you know, that’s how I got into it. I got into it because of like this entire, like cultural moment that happened.

Jeremy Nelson (03:11):
Right, right. Yeah. It was really transformative in some way. I mean, it went rather quickly, but I think it really highlighted what’s possible for so many areas. Right. I’ve heard a number of folks talk about how it opened our eyes.

Joanna Millunchick (03:23):
Yeah. Yeah. Because before that VR struck me, you know, VR in particular struck me as being, you know, like just the realm of gamers and, you know, it was kind of, it would cut people off because you put this giant thing on your face and you’re cut off from the rest of the world, but you know, this particular game. And then, you know, all of the things that were done subsequently with, with Snapchat kind of showed me that there are other modalities for, you know, mixed reality. And how people can use them, especially since they’re just on the cell phones. Right. You know? So that, to me, it was just like, that was like, Oh, I get it.

Jeremy Nelson (04:00):
Yeah. That’s great. So then you brought that to the classroom or where did they, where did that go next?

Joanna Millunchick (04:10):
Well, so the next thing that I wanted to do was, you know, figure out how, how to actually make this happen. And I was able, I found out there was a student, a group of students actually, who were trying to make augmented reality work. They started a company and they were making like augmented and virtual reality games for kids who are sick in the hospital. And so I started talking to them and talking to them about the idea of like, I wanted this to be an augmented reality to begin with. Cause I thought it was important for first of all, for the objects to be situated in real space as opposed to virtual reality. And I also thought it was important that we use people’s cell phones. You know, because another piece of it is, you know, I didn’t want the students to have to get all this extra technology.

Joanna Millunchick (05:00):
I wanted to use like ubiquitous technology and I still feel that way. And you know, that sort of started us on our journey. The company name is Gwydion it no longer exists, unfortunately, but you know, we started working on it for a couple of years. We started with the augmented reality first because of the reasons that I just described, but we found that there were some technological issues that we run into. And so we pivoted to virtual reality, not because it’s necessarily a better solution, but there were other things that needed to be solved before we could pour it into the augmented reality space.

Jeremy Nelson (05:41):
Yeah. I think it’s advanced, you know, Apple and Google have done a lot with their ARKit and ARCore. And I think they’re laying a deeper foundation these days to really make that more viable.

Joanna Millunchick (05:51):
Yeah. You know, and the thing that we’re really learning is that, you know, you put this in the hands of students, I was kind of assuming that it would be easy. Right? You know, cause this group of students they’re like digital natives and you know, they will just get it. But you know, what’s really interesting to me is that, you know, the normative context is completely different. So what do I mean by a normative context? So like if you walk into, you know, fast food restaurant, all of them are the same, you know, you know exactly what to do. You walk up to the counter, you order and then you stand back and you wait for your food. Right? Everyone knows, everyone’s been trained. You know, how do you, how do you read a book? You know, you open it up and you go from, you know, you, you turn the pages and so forth.

Joanna Millunchick (06:36):
How does the mouse work? We all know how that works, but you know, there’s no reason for us to know how to do that. We’ve been trained the normative context in virtual reality. Isn’t there yet? You know, there’s no like single way of interacting with these controls. And it seems like every app does things a little bit differently. And so you put the controls in the hands of the students and they’re just like, what do I do here? So there’s a large learning curve. And so that was something that I wasn’t expecting, you know, the normative context, you know, in establishing that normative context and having like a really good user interface. I know the designers out there are going, duh, but you know, that was a hard learn a though basic lesson that, that we had to learn.

Jeremy Nelson (07:22):
Yeah. No, I mean, there’s still a lot of experimentation going on. There’s still isn’t great best practices or the research on where, you know, where the interactions or the affordances make the most sense or work better. Well, so did you, did you go down the path, you brought it to VR to a classroom or how did you take it from there with Gwydion?

Joanna Millunchick (07:44):
Yeah, so we started, you know, we have a research project that we’ve been working on for past couple of years now. And so we, we got it to the point where we can actually get it in the hands of, of students. So there’s a couple of different things that we did. So to start with, we just started with a few people, we had eight people just sort of play around with it. And we, we design these learning activities where we asked you to look at, you know, like a a crystal structure, either on paper or in virtual reality. And we just asked them to like talk through their thought process as they’re, as they’re trying to solve, answer these questions that we’re posing. And, you know, one thing that we immediately saw was that students who were on paper, working on paper, they’re like, Hmm, I don’t remember how this goes.

Joanna Millunchick (08:32):
Right. And they were trying to do things from memory, but students, even if they had like zero experience with the, with the concepts before, could were like manipulating the structure and going like, Oh, look, I can see, I can see this thing here. And I can see that if you look at it in this perspective, I can see that the atoms are touching. And if I turn it around this way, it doesn’t look like they’re touching, but you know, so, so they actually talk through what’s going around, whereas, you know, what the students were doing, you know, on paper, they’re trying to remember what it was that someone said about the structure. And so I thought that was an interesting initial observation. So we kept going and we modified the learning activity. And the next year we gave him like this inventory, like concept inventory, where they were just testing themselves, you know, just to see what they knew ahead of time.

Joanna Millunchick (09:26):
And what we, what we found was that, you know, there are certain structures that you have to like turn in your head in order to understand what the relationship of the different atoms are to one another. And we found that students didn’t understand how to do that super well, but if they saw the structure in VR and could turn it, then they could answer the question appropriately. Right? So if they could actually physically turn it, then they answered the question. So that was very exciting. And so then the next year we tried, tried it again with a different class and we wanted to see whether or not that benefit was retained and it wasn’t, you know, so we saw, we saw again that the VR helped students answer the question correctly, you know, if they could take the structure and turn it like in, in virtual reality they could, they answer the question better than those who just did it on paper. But then when we asked them about, you know, similar kinds of questions later, they didn’t, they couldn’t do it as well as the people who had did it on paper, who did it on paper. And so I was wondering so now the question is, is why is that?

Jeremy Nelson (10:44):
Right? That’s, that’s what I’m thinking of. Why? What about that? Do you, do you have any hypotheses or have you tested it yet?

Joanna Millunchick (10:52):
Well, I have a hypothesis and we’ll see whether or not it pans out. I think it’s kinda like using training wheels, right? So if you put a kid on a bike with training wheels, they can ride but then you take the training wheels off. It doesn’t mean that they can necessarily ride it without falling over. Whereas the students, you know, the, the kids who never had training wheels, you know, maybe they can, you know, cause they, they kind of struggled through it and they figured out how to balance without the training wheels. Maybe they’re going to learn faster. So I don’t know, maybe, maybe VR is acting as training wheels. I don’t know. You know, and is that a bad thing or is it a good thing? And so, so I mean, it’s a fascinating question about learning, right? Because it’s like, okay, so we gave them this advantage, if you will. And you know, maybe that’s a bad thing or maybe we need to learn how to use that appropriately so that they can learn more quickly. So I don’t know. It’s just, it’s a, it’s a fascinating question. U

Jeremy Nelson (12:07):
Right. What do you think it’s like similar to, like, if you give, teach students with a calculator early on, right. A calculator, how to do math versus, you know, the traditional writing it out and kind of that repetition and knowing how it works. Is it somewhere in that realm? Because, you don’t learn with a calculator right away, because you can just burn, burn through everything. You don’t really understand how it works. Maybe, maybe it’s something in that realm.

Joanna Millunchick (12:34):
Yeah. And so I, I don’t know the answer to the question yet, but you know, absolutely. You know, it seems like, it seems like it does change the way students interact with the material, which is a good thing. I mean, we, we do want to see that, how do we, how do we incorporate the virtual reality in a way that helps students learn more quickly? You know, and so maybe, and maybe the answer is, is that, you know, this isn’t something for introductory students and really what we need. We need them to struggle with the easy things. Maybe what we ought to be doing is using these kinds of tools in more advanced classes where they’ve already kind of established, you know, that they understand how to do the spatial reasoning and help them, you know, do more complicated kinds of things. So as opposed to, instead of training wheels, this actually becomes, you know, a different kind of tool.

Jeremy Nelson (13:40):
Sure. I road bike or mountain bike.

Joanna Millunchick (13:43):
Yeah. Yeah.

Jeremy Nelson (13:44):
Yeah that’s cool. That’s interesting. Well, you, you sit in a unique position here at the university with your role at the college of engineering, like have you seen, have you seen other examples of XR that, you know, are good examples or experiments? I mean, what have you seen in that position and what are you interested in?

Joanna Millunchick (14:05):
Yeah, you know, I was super lucky because just as I was getting more and more interested in the VR and XR space I stepped into the associate Dean role and the Dean asked me, it was just like, so what is it, you know, like if you could do anything with this, with this role, what, what is one of the things you’d like to do? And I’ve always been kind of like a, a tech nerd, like, you know, show me new technology and I want to start using it as quickly as possible. Right?

Jeremy Nelson (14:34):
I can relate.

Joanna Millunchick (14:36):
Yeah. You know, and I just always wanted to mess around with this stuff. And so when I stepped into this space, I wanted to enable other people to do that too, especially in VR. And so we established this this mixed reality clinic, the Duderstadt center already has, you know, visualization lab, lots of facilities and lots of people who really know their stuff. But you know, what was always missing is that, you know, we didn’t really have a way for faculty, I think, to get an idea and be like, okay, I just want to implement my idea without having to be an expert in unity or developing this thing.

Joanna Millunchick (15:18):
So when I stepped into the role, I created this mode to allow faculty to experiment with mixed reality. We hired a person whose role was specifically to talk to faculty, understand what the project was and advise them on what are the best tools that are available to get it done. And if the tools don’t exist, then hook them up with resources in the college or in the Duderstadt center to get their stuff done. Another piece of this is we also are working with a postdoc in School of Education who knows how to actually do research, you know, cause there’s, there’s two pieces to this, you know, cause we want to do the developing, but we also want to make sure that what we’re making, we just don’t want to make stuff and throw it over the wall. We want to make stuff that actually, you know, has a difference, makes a difference in, in learning and in teaching. So, so that’s, that’s been, my focus is we don’t just want to develop things. We also want to make sure that what we’re doing actually helps students and how does it help students and so forth so that we, you know, cause we can spend a lot of time developing this thing and then find out like I did that actually you don’t want to be using it in this context. This is not the best context you want to be using it in a different context.

Jeremy Nelson (16:39):
Yup. Yeah, no, we’re aligned the same way in the XR initiative to make sure we understand where it’s most effective and how to, how the students learn better.

Joanna Millunchick (16:48):
Yeah, yeah, yeah, no, I think that, you know, we were all starting these things up at the same time and the conversations in the university were really similar in that regard. You know, wanting to make sure that, that, that was the focus. So yeah. So I’ve been, you know, we’ve been able to fund a number of these initiatives. The first two that we funded one was very similar to what I’ve already described. You know, how do you, how do you actually visualize these different kinds of structures, but the other one was about how do you train students to do like like customer discovery? You know, so an important part of design is understanding who you’re designing for. What does that have to do with mixed reality? Well, there’s different kinds of people that you can design for. You know, you can be talking to, you know, a middle aged white dude, or you could be talking to children, you could be talking to you know, the Hispanic community for instance.

Joanna Millunchick (17:54):
And so what they did is they created a virtual reality kind of like a simulation, if you will, where, where students can interview people with different kinds of identities. And so it’s a way for them to interact with people in kind of a safe space. Cause otherwise what you’d have to do is you have to get, gather all these people together and allow students to practice on them. So, you know, that’s like this, whereas what I was doing is I was trying to get students to interact with three dimensional objects. This is trying to get people to, you know, students to interact with different kinds of people to build up empathy. So that was an interesting kind of approach. The, the projects that are being funded this year, you know, there’s one about earthquakes and, and how do you actually interact with the community around, you know, where an earthquake has passed through, you know, so that’s an example.

Joanna Millunchick (18:55):
There’s another example. So that one is, is it’s kind of a combination between like interacting with objects and also interacting with the community. So it’s like the spectrum, like on one side of the spectrum, I’m finding you can be interacting with objects. And then at the other end of the spectrum, you’re interacting with people to develop empathy, you know, so it’s that entire spectrum of interacting with objects and trying to understand, you know, spatial reasoning kinds of things all the way through to interacting with individual people and developing empathy and everything in between. That’s kind of how the different projects are.

Jeremy Nelson (19:36):
Yeah, no, that’s, that’s great. I, I I’d seen the project about the customer discovery and it got me thinking about lots of other ways to use that type of technology for say, law students and practicing giving their oral arguments or, you know, business students giving a pitch or negotiate. Like there’s, there’s other ways to apply that underlying technology and in an immersive way they use natural language processing, you know, AI

Joanna Millunchick (20:05):
All of that. It’s so interesting. Yeah.

Jeremy Nelson (20:10):
That’s, that’s super cool. Well, in terms of, you know, concerns, what sort of concerns do you have about the future of this technology or how we help shape it?

Joanna Millunchick (20:21):
Well, you know, I kind of commented on it in the very beginning, you know, so one thing having to do with the normative context, you know, we, I think that, you know, as a technology, we need to figure out, you know, what are the best ways we have to kind of agree on, you know, what these technologies are going to look like and how do you interact with these technologies? I think normative context is important thing, you know, having you know you know a kind of file structure that you can import things in, you know, how is it that you can get individual students to interact and build their own things? I, you know, there are, there are technologies that exist for that. But it’s not like, you know, a word document or a PDF, right. We’ve kind of like rattled down into a way of doing things and we can share these, these documents these objects, if you will, in, in a broad way, we don’t have that yet for virtual reality objects or augmented reality objects.

Joanna Millunchick (21:30):
So I’d say that’s a thing that we need to think about. That’s, that’s what we kind of spend most of our time doing is like, how do you, how do you just make, and then making the content, right, right. We’ve, we’ve got books and books and books and books filled with content, but how do you take that content and translate that into virtual and, and, you know, mixed reality. I mean, we don’t have libraries of virtual content yet. And making that content is so hard, right?

Jeremy Nelson (22:05):
It’s complex.

Joanna Millunchick (22:07):
It is complex. And do you have to, for every single thing that you want to do, do you need to build a whole new app? Or is there like a way that we can, I don’t know. It’s just, we haven’t figured that out yet. So that’s what I would say is like the, from my point of view, it’s how do we get the content into virtual reality?

Joanna Millunchick (22:28):
And then how do we, how do we come up with a way for people to interact with that content in a way that is easier than it is now, you know, where we don’t have to train people like here, if you put your finger here and you, yeah. So that’s, those are the kinds of things, you know, I know lots of other people, lots of other people are worried about things like security. Obviously these are important things. So I, I, I don’t mean to diminish, you know, the importance of those things, you know, you asked me and the thing that gets in my way, the most is content development.

Jeremy Nelson (23:07):
Yeah. Yeah, no, I mean, that was one of the things I, I saw early on and we did a similar XR innovation fund and funding project to see how can we create content faster? How can we leverage, you know, best practice processes or other platforms. And there’s a, there’s a lot out there and people are really exploring it. And it’s not, it’s not as mature as some of these other platforms that have been around for 20, 30, 40 years even. Right?

Joanna Millunchick (23:36):
Yeah. But it’s funny because actually VR has been around since the early nineties. Right? I mean, the technology has been around that long, but for whatever reason, you know, I think that the computing power just wasn’t there in the early days, you know, and the ability to, you know, like store so much data wasn’t there in the early days. You know, and now we’re finally getting to a point where it is a lot easier and it’s a lot cheaper and I think we’re, we’re close, but you know, people have been working on this for decades and still haven’t figured it out.

Jeremy Nelson (24:13):
Yeah. Well, let’s lead the way. Okay.

Joanna Millunchick (24:17):
Let’s do.

Jeremy Nelson (24:18):
Right. Yeah. In that context, what do you want to see us do here at Michigan to enhance education? I mean, you’re already doing it, you’re helping lead the way and in your role, the college of engineering, but more broadly, what would you like to see Michigan do?

Joanna Millunchick (24:31):
Well, I think that the current situation, you know, the COVID crisis is kind of like a, it’s a mixed blessing, you know, on the one hand we were going in this direction anyway, you know, trying to get these learning technologies in the hands of students and you know, what this situation has, has really shown us. You know, it’s just accelerated that timeline. You know, if we can get, you know, mixed reality into the hands of students more quickly, you know, can they learn, can we get them to interact with these materials and not have to be in a class? That would be awesome. So, you know, where I would really like to see Michigan lead is in coming up with the, you know, important ways in how mixed reality can be used as not just entertainment, but also a learning tool. And I’m particular particularly interested in the STEM fields, just because of my training. But, you know, it’s not just about STEM. It’s also about, you know, literature and the humanities and dealing with race and dealing with different kinds of identities and so forth. So I really do see a lot of places in which mixed reality can be incredibly powerful as a learning tool. And I want us to be, I want Michigan to be, you know, the place where it happens, where we are, the ones we are, the leaders that everyone else looks to in this space. So that’s what I, that’s my wish.

Jeremy Nelson (26:12):
All right. I’ll accept the challenge.

Joanna Millunchick (26:15):
Okay. Challenge accepted.

Jeremy Nelson (26:17):
Yeah. That’s great. Well, this has been really interesting. I, I learned more than I had in our previous conversations about the start of your work. I didn’t, I didn’t realize it was Pokemon go. That’s great.

Joanna Millunchick (26:29):
Yeah, yeah, yeah, totally.

Jeremy Nelson (26:32):
You know, you’ve heard some of our other podcasts, you know, what other topics should we be exploring? Who else should we be talking to? What would you like to hear?

Joanna Millunchick (26:40):
You know, there are so many experts that I am loathed to name just one. I mean, every time we, we get a group of people together to talk about mixed reality, I am excited and kind of embarrassed that I don’t know everyone in the room because it seems like new people are constantly coming into the room. You know, like I’d like to hear a little bit more about what’s going on in medicine. I know that, you know, the medical field is, is one of the biggest places where they’ve really embraced these technologies for training. I think that is just amazing. I would love to hear from Sara Blair and the things that she’s doing around using virtual reality and mixed reality to look at literature in particular, that is so amazing. And I know that Lisa Nakamura is looking at using VR to examine identity and race, which I think is just fascinating as well. So things that are like have nothing to do with STEM, I would love to see that. I would also love to see what, like the math department might be doing around using virtual reality or chemistry, you know? Cause you know, so those are some of the areas that I, I would love to hear more about.

Jeremy Nelson (28:01):
All right, I’ll put them on my list. The math one was interesting. I was, there was this new VR game called Alex half-life and in VR and somebody had taken one of the scenes in the game and like started teaching math in VR in the game.

Joanna Millunchick (28:22):
That is so awesome.

Jeremy Nelson (28:23):
Right? Like fascinating, reminded me of that when we did the hybrid immersive teaching symposium, the somebody from somewhere up in Duluth, they were teaching med students about communication and leadership with this VR game called Keep Talking and Nobody Explodes.

Joanna Millunchick (28:45):
Oh my god that’s so great.

Jeremy Nelson (28:45):
Yeah. So it’s, it’s a game where somebody is in VR and people around them aren’t and they’re trying to communicate with what they’re supposed to be telling the person to do, but they can’t see what they’re seeing. And so they have to have this dialogue and they have to communicate. And so they were trying, they were using this game to teach them communication skills for med students, as they’re communicating with different people. And I thought that was just a very interesting use of a game and in a way, I’m sure the creator did not intend.

Joanna Millunchick (29:11):
Right. No, I love that. I think that’s just great. I think that’s amazing.

Jeremy Nelson (29:16):
Yeah. Well, thank you so much for your time. It was great to talk with you and learn more about your work and where we’re going and appreciate it.

Joanna Millunchick (29:24):
My pleasure, always a pleasure.

Jeremy Nelson (29:32):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.