Jeremy Nelson, Director of XR Initiative

In this week’s MiXR Studios podcast, we talk with Mark Newman, professor of information at the University of Michigan School of Information and a professor of electrical engineering and computer science in the College of Engineering. Mark was instrumental in the early days of XR (called AVMR at the time) at the university. He led the charge to create a new XR Graduate Certificate program. This journey started with a trip to Seattle around 2017 to explore what Microsoft was doing in the mixed reality space and to visit Oculus.

Starting in fall 2020, graduate students at U-M can earn a graduate certificate in Extended Reality. The program is administered by the School of Information and already includes students from the following schools:

• Stamps School of Art and Design
• College of Engineering
• College of Literature, Science, and the Arts
• College of Nursing
• School of Dentistry
• School of Education
• School of Medicine
• School of Kinesiology
• A. Alfred Taubman College of Architecture and Urban Planning
• School of Music, Theater, and Dance
• School for Environment and Sustainability

We discuss the focus on students, and how we can bring more people into XR to be the creators of the future. XR technologies really bring together multi-disciplinary teams to create compelling content. It speaks volumes that more than 60% of the schools and colleges at the university are participating in the XR graduate certificate from launch. There will be many opportunities to develop the next generation of XR builders, and we will combine that with many courses where XR is the object of study or the delivery mechanism for content. We explore how important it is to have faculty in our School of Education who are studying this technology, and how we can help students learn in new more efficient ways.

Professor Newman grounds the conversation in the reality that the hard work now begins. We must understand how best to use XR in teaching and learning. For all emerging technologies, there is a hype cycle, and virtual reality, in particular, has been through a number of cycles. The mass production of devices is providing new opportunities to help sustain beyond the hype. We need to strike a balance between experimentation and creating content and software that can scale across programs and schools.

I had a great time talking with Professor Newman that helped me understand better how he helped shape the XR curriculum here at Michigan. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 12

Jeremy Nelson (00:09):
Hello, I’m Jeremy Nelson, and today we are talking with Mark Newman who is a professor of information at the University of Michigan School of Information. We are talking about his work and bringing together the XR community at the university and the creation of an XR graduate certificate. Coming up next in our mixr podcast.

Jeremy Nelson (00:44):
Welcome Mark. Thank you for joining us today.

Mark Newman (00:47):
Thanks for having me

Jeremy Nelson (00:48):
Yeah, we’re super excited to talk with you today and learn more about all of the foundational work you did to bring that XR working group and committees, and ultimately the, the graduate certificate program. Would you, would you mind sharing a little bit about, you know, just how everything got started, how you got involved and what brought us to this point?

Mark Newman (01:12):
Sure. So I’m trying to remember exactly what it was. It was, it was probably 2016 or 2017 probably, right. Actually that, that school year there was a lot of activities starting around, you know, trying to build more of a XR or, you know, as we were calling it at the time, AVMR, augmented virtual and mixed reality kind of a more coherent campus wide you know, initiative, there’s a recognition that there’s lots of stuff there, there was a lot of stuff going on around campus. But we weren’t talking to each other. And we weren’t you know, putting ourselves out there, you know, we, we kind of recognized that we were actually probably ahead of what a lot of other universities were doing. This was around the time that industry was starting to you know, really put some serious investments into XR you know, the HoloLens was just getting off the ground.

Mark Newman (02:20):
The you know, the Oculus rift had, had pretty recently launched and, you know, there were things like Magic Leap had been announced, but that was still quite a ways away. And so actually originally Rebecca Pagels and Amy Klinke, so Rebecca Pagels is does development work in the School of Information and Amy Klinke is in the business engagement center, I think in engineering. They sort of saw this as an opportunity to try to coordinate a lot of the stuff that was going on. And then, you know, initially go out and try to find some funding and some resources mostly from industry to try to get something, you know, kind of more substantial and coherent that way. And so I was Rebecca, I think originally reached out to me as somebody who’s just kind of working in my research and teaching has always been in sort of emerging technologies in a sort of mobile and ubiquitous computing.

Mark Newman (03:21):
And I’ve always been interested in particularly augmented reality. That’s always been kind of, you know, part of this ubiquitous computing vision that’s inspired me since the beginning of my career. And you know, starting to see signs that, that augmented reality and particularly not sort of the augmented reality of you know a staged environment, you know, that’s like a, sort of a cave type situation, but, but AR in the real world, like out mobile AR Google, Google glass was also kind of, you know, getting a lot of hype at that, at that point. And even though the Google glass had a lot of shall we say shortcomings the vision for me was very compelling, you know, it remains actually very compelling, you know, and, and I’m still waiting for somebody to do that right

Mark Newman (04:19):
Because I think that’s very exciting when we get to the point where, you know, we can have kind of this layer on top of the lived, the built environment and the world that we live in, you know, that’s providing just-in-time information and then, and you know, things that are relevant to our goals and needs and so on and so forth. So anyway, this was all happening. And I was, you know, very excited, very interested. And Rebecca approached me and said, ‘Hey, would you like to be involved in this?’ And I said, absolutely. You know, I want to see where this, where this is going. And so I was part of an initial group that Rebecca and Amy pulled together included Oh Michelle Aebersold from nursing. Prashant from medicine sorry, I’m drawing a blank on his last.

Jeremy Nelson (05:08):
Mahajan?

Mark Newman (05:09):
Mahajan, that’s that’s him. Yeah. who else was involved? Pete Bodary from Kinesiology. And some folks Joanna Millunchick and, and, and, and anyways, just kind of the people who were, you know, starting to coalesce around this. And we actually did a field trip. We went out to we went out to Seattle and we visited Microsoft and we met with the HoloLens folks and had some discussions there. And then we dropped in on the Oculus offices in Seattle and got some pretty awesome demos and chit chatting with them. And so that was kinda how it all started. It really started with, you know, trying to come up with a coherent you know, pitch and, you know, an attempt to get some kind of serious some serious resources. And also thinking about how do we, you know, establish a leadership position for the University of Michigan.

Mark Newman (06:03):
You know, we saw that we’re doing all of these things. And I think one of the things that’s really interesting about the University of Michigan you know, when you compare us to other places that are doing exciting work in XR is that our interest in XR is across campus. Right. So think about the people that I just listed. Right. You know, we got people from nursing, we’ve got people from medicine, we’ve got people from kinesiology, you know, since then we’ve added architecture you know, landscape architecture and the school of the environment. We’ve got folks from education you know, all over, all over campus. And, and, and when we talk about the certificate in a minute, I’ll, I’ll give you some numbers on that. But so that, that’s really exciting because we’re not just, you know, trying to push the envelope on the technology, you know, not just trying to, you know, make you know, faster graphics processing, or better vision wrecking, you know, computer vision systems or things like that.

Mark Newman (07:05):
We’re actually looking at like, where does this technology you know, where can it make a difference? What are the applications? And, you know, where is it a real game changer for things that are important for, for humans, you know, for like, how does it improve our health? How does it improve the way that we train our health practitioners? How does it train, how does it, you know, improve the way that we educate K through 12 students, you know, how does it improve the way that we educate college students and things like that. Right? And so this is all very exciting to me. You know, my background, you know, as I mentioned before, is in emerging technologies, but specifically in the area of human computer interaction, you know, so how do we design systems that people can use and that serve human needs.

Mark Newman (07:50):
And I come from a computer science background, so I have, you know, the technology sort of background and focus, but my interest has always been in, you know, how do we make this, you know valuable for for things that people really want to do and need to do and so on and so forth. So this was all very exciting. And so we had those initial conversations and, you know as often goes with these kinds of things, you know the corporate corporations were like, yeah, this is really exciting. We’d love to help. And then, you know, what does that actually mean? It took a little while anyways, still, still kind of shaking out you know, but definitely a lot of interest. Right. and we’ve, you know, since then talked to many other industry partners as well, and had gotten a lot of interest, and we’ve gotten some, some really engagement really engaged with them, some of them, but one of the things that came out of that was I, I hope I’m getting the dates, right.

Mark Newman (08:48):
It’s all a blur you know, time is kind of an abstract concept at this point. Might’ve been yesterday, or it might’ve been, you know, four years ago, but the 2017, 2018 school year, I believe. Yeah, that sounds right. What I was asked by Dean Finholt, Dean Tom Finholt of the School of Information to chair a campus committee that was called at the time the AVMR steering committee. And so this was kind of the first coherent group on campus of the, you know, trying to bring together all the people from diverse departments and units and schools and colleges that were interested in this. And we were asked to do three things. One was to to actually write a white paper about summarizing all of the research that was going on on campus and kind of articulating a vision for where, you know this type of where AVMR research at University of Michigan, like where, where we could go.

Mark Newman (09:55):
And, and what, what were the sort of places of synergy what were the challenges and what would it take for the university to support this kind of effort, right? Like what, what sorts of institutional support, whether that be resources that be, you know, better ways of connecting across units and so forth. So that was one, one challenge. One thing that we were charged with the next thing that we were charged with was to try to get a better understanding of the current resources, like, you know, actual like equipment, you know, like we knew there were, you know, dozens of HoloLenses, for example, around campus, but everybody had gotten their own sort of like private stash from Microsoft and they were using them for whatever, but, you know, probably sitting idle most of the time. So how do we kind of coordinate that and you know, in particular, how do we leverage the existing institutional structures already on campus?

Mark Newman (10:51):
You know, like the libraries and the do just add center and those kinds of things could make it so that we could get more people involved. We could provide resources, we could support students and faculty doing all this kind of work. And then the third thing that that committee was asked to do was to put together a proposal for a graduate certificate. And so certificate, you know, here at the university is basically is it a, you know, a small educational program. So they typically are nine to 12 credit hours, which basically means three to four courses that anybody, any graduate student on campus more or less can pursue sort of in parallel with whatever their main you know, focus is. So if you’re doing a, whether you’re doing a, you know, a master’s degree in the School of Information, or you’re doing a PhD in English, or you’re doing, you know, a master’s degree in computer science you can take on one of these certificates as kind of an adjunct to whatever you’re doing and allows you to focus in a particular area.

Mark Newman (12:00):
And so we started talking about the, you know, how to do this and what the focus should be and so on and so forth and ended up with having 12. So I might have these numbers not quite exactly right. So I think there are 19 top level units at the university schools and colleges and we had 12 of them involved in this committee that were involved in and interested in participating. And I think since then we’ve actually added another one or two. Yeah, so, so that’s first of all, for me, this was kind of my first experience doing sort of, you know, leading a campus wide group or even actually being involved in a campus wide group. I’d never done any committee work outside of the School of Information. So that was fascinating learn about how things work around the campus.

Mark Newman (12:55):
And then I learned a lot more when I had to get letters of support from all 12 deans. So that was all interesting, but so so that we put together this proposal and then, you know, I will, I’ll tell you more about how that kind of landed in minute, but that was the early sort of history of you know, how this initiative came to be. And then since then, you know, many other initiatives kind of have sprung up and, you know, some grew out of that and some happened parallel and it’s converged and so forth, including the initiative that you now lead. And so that’s, that’s kind of the, that’s the, the, the history the early history of how, you know, some kind of organization around AVMR or what we now call XR came to be.

Jeremy Nelson (13:47):
Yeah, no, I mean, that was, that was one of the things that drew me to the position is just that broad interest across the university, the collective efforts that already had been occurring, you know, as I’ve looked at other institutions, you know, things may have lived in an art department or computer science department, and there’s no one group looking at it and trying to expand out, but there was already a broad foundation here in the interdisciplinary work occurring that I was like, Oh, there’s so much to work with here and really to help accelerate and take it to them.

Mark Newman (14:20):
Yeah. It’s incredible. Isn’t it?

Jeremy Nelson (14:22):
Yeah. It’s great. It’s great. So yeah, so from there what was the decision for going for a graduate certificate versus undergrad minor or major? What was the thinking behind that?

Mark Newman (14:39):
Well, one of the main considerations is that it’s a lot easier than any of those things. It doesn’t it’s, it’s governed by Rackham, which is the sort of the unit on campus that pretty much all graduate students are enrolled in. So you only have to talk to one unit, whereas like an undergraduate minor or something like that usually at least as I understand it, you know, those would live within, you know, one particular unit like LS&A, or, or something like that that may not be entirely true. There are cross unit minors but it’s, it’s a relatively lightweight process. And so that’s one, one consideration is that, you know, I think there are actually interested in developing some of these other programs. But this one would be, you know, kind of a way to get things started.

Mark Newman (15:34):
And one of the, you know, so some of that is, you know, we wanted to provide opportunities to students who were interested in this, right. Obviously, you know, and so you know, attract students that you know, so we’ve actually already had students you know, just like this last year who were accepted into programs in like architecture and the School of Information who said that the XR certificate was something that they considered when deciding where to go to pursue their graduate studies. So that’s that’s yeah. So, I mean, we’re, we’re starting to see that, you know, we’re already, even though the thing hasn’t officially launched yet, it doesn’t launch until the fall you know, that, that, that just even being able to put something out there and get it on people’s minds has been helpful. But the other thing is that, you know, it’s really helpful to have something, you know, that’s kind of a concrete academic program, you know, or it could be like a research center, or it could be something like that that is cross unit that, that people are formally associated with and participating.

Mark Newman (16:40):
And then there’s, you know, it’s actually building something and running something to keep those lines of communication, open to foster things like interdisciplinary research to foster things like the development of new curriculum, you know, so like new courses that might be cross-listed across units and things like that. And, you know, also just to get faculty to think about, you know, developing new courses and developing new curriculum, if you have something that you can point to and say, okay, I want to do this, get this on the books. You know, and it can be part of this certificate you know, that can help to navigate each units, you know, curriculum committee and, you know, the approval process that they have, you know, getting the Dean to be interested in and actually say, okay, yes, you can teach that course instead of, you know, the intro 101 course that

Mark Newman (17:32):
We previously had allocated to you. And so all of those things I think have been been really helpful. So that was the you know, those were essentially the considerations. And then, you know, the other part of it of course, is that you know, the industry partners that we were talking were encouraging us to do this, you know, so they’re, they’re, you know, they’re interested of course, in working with us on research and developing new applications and new technology, but what they’re really interested in, probably even more so is hiring our students and they want our students to have the skills that they need. Right. And so these companies that are, are starting to move and are not, not starting anymore, but that are, you know, have have significant investments in, in XR, so these companies, you know, they need, they need people that can step in and start to work in these environments that understand what it is, understand what the potential is, and now of course understand how to work with the technologies, how to produce content, you know, how to you know, work with the emerging you know platforms and, and, you know, these kinds of things. So that was also a major motivation was that this was what industry was telling us they wanted us to do. And of course we wanted to do it. So that’s a pretty good publicity.

Jeremy Nelson (19:05):
Right, right. Yeah. And in my discussions with some of these corporations as well, they’ve, they’ve echoed that. And, you know, we’ve, we’ve hired two XR developers in our XR initiative to help create processes and protocols for faculty that want to engage in XR that, you know, they can consult with us. We can either help them source content or build content. And what we’re looking to do is bring students in and be fellows within our program, learn the skills to help develop these experiences, project management, 3D modeling developing with unity or unreal or WebEx are. And then, so they’ll be better prepared. They’ve worked on something that’s applicable to maybe them or, or classmates, and that they’re better prepared. They’ve used industry tools. So when they go out into the field they’re ready to go. So,

Mark Newman (19:55):
Right. Yeah. And of course, you know, one of our goals, especially when thinking about, you know, graduate education is, you know, we’re, we’re, we’re training students, not just, you know, obviously they need to be conversant with the current tools and, you know, with the current and the current platforms, but to also have a deeper understanding of kind of the dialogues that have been happening, you know, actually for, you know, decades around virtual and augmented reality and what the potential is, what the vision is, and to understand, you know, the limitations of current technologies and all, but, but how they’re evolving towards, you know, what their ultimate potential could be. So I always like to think that, you know, we’re training our graduate students, not for their, you know, not for the job that they’re going to get right when they graduate, but for maybe the job after the job, after the job that they’re going to get. Right. And so really trying to the future leaders in this field and not just, you know, the people that can, you know, get something off the ground, you know, six months from now, but, you know, can actually chart directions that the industry is going in the longer term.

Jeremy Nelson (21:05):
Yeah. Yeah, no, I think that’s, that’s great. And one of the areas we’re most interested in is how do we, you know, assess or evaluate, you know, where these technologies can help in the learning experience and what sort of interactions, what sort of experiences will actually aid in students’ learning where versus, you know, a traditional method or video or something else is equally as good or better. So we want to make sure we have best practices out of this, of like how to deploy the technology to support learning.

Mark Newman (21:37):
Yeah. And I mean, just to, you know to, to send some praise back in your direction, I mean, I think the stuff that you guys are doing in that regard is fantastic, you know, really trying to actually put stuff into the classroom and then, you know, like systematically, I mean maybe rigorously, systematically might be ultimately we want to get there. It might be overstating it in the near term, but, you know, it’s like really be reflective, I guess, about, you know, what’s working and what isn’t, what’s difficult to do. What’s easy to do, what do we need to do? And, you know, I love that you know, folks like Chris Quintana in education and Michelle Aebersold in nursing are, you know, really thinking about this education, you know, what is the value for education? And that’s something I think that we can do it at Michigan extremely well. And I’m really excited that we have, you know, such great folks that are looking at this and really trying to understand exactly what you said. Like what, how do you know, what are the benefits of them? What are the practices that we need to follow to achieve those benefits? And, you know, ultimately what tools and practices can we develop that will facilitate using these technologies more effectively for, you know, things like education and these other domains as well.

Jeremy Nelson (22:55):
Yeah, no, it’s great. At any concerns you have about the future of XR for teaching and learning or areas for us to be conscious of?

Mark Newman (23:04):
You know, I mean, I think I mean, my concerns are mostly the, have to do with navigating you know, the path. I mean, I think, you know, we can look at something like a PowerPoint, you know, which has been you know, criticized heavily you know, in many, in many domains for, you know, putting, you know standardizing the way that, you know, information is presented in a way that hasn’t always been healthy or productive. And I, you know, I think that argument has some weight, but I also think that people have found ways, you know, to use tools like PowerPoint, like, you know, a thoughtful and reflective educator, you know, can, you know, find ways to use the tools to accomplish what they need to accomplish and things like that. And so I suspect that, you know, as XR, you know, as people get more excited about using XR in education, you know, people will use it in, you know, frankly kind of dumb ways, you know, sometimes, but I don’t think that’s, I don’t think that’s damning of the technology necessarily.

Mark Newman (24:12):
It’s just part of the learning process.

Jeremy Nelson (24:14):
Exploration.

Mark Newman (24:15):
Yeah. I mean, you know, people can use any tool and dumb ways. You can use paper and pencil and like pretty dumb ways. Right. So so, you know, a lot of it has to do with just, you know, where do we get there and, and, you know, sort of the you know, the the hype curve, right. And you know, I haven’t, haven’t tracked, you know, where a Gartner puts XR you know, on its type curve recently, but you know, definitely, you know, it feels to me like, you know, 2017, what you know, are you familiar with the Gartner? Hopefully our listeners are, they can, they can look it up, but you know, I feel like 2017, at least for XR and education, you know, we were at the peak of inflated expectations.

Mark Newman (25:03):
Right. You know, like, Oh, transform everything. And there were a few great, you know, demos you know, so our provost at the time, you know, had this sort of famous you know, almost religious experience you know, when he got a demo from, I forget the name of the local company who had that really great demo of like molecular structure, I think, you know, sort of VR approach and, you know, that, you know, led to kind of the enthusiasm that has created a lot of the you know stuff that’s going on around campus. And so, you know, like as is often the case with new technologies, you know, you can find a really inspiring example kind of a low hanging fruit, you know, pretty, pretty early on. Right. And that’s great. That’s important. You have to do that in order to get people excited about it and see the potential, but then the hard work starts, right

Mark Newman (25:53):
Of really just slogging through and figuring out, okay, how do we scale this and how do we, you know, make it accessible? And how do we, you know, where does it make sense? And where does it not make sense? And, you know, where do you, where does it provide something really substantially, you know, that, that, that justifies the investment and all those kinds of things. It feels like we’re, you know, we’re at that point now where we’re doing the work and I think you and your team is really, you know, doing that work of kind of saying, okay, well, you know, how do we get to a, I forget what they call it, at Gartner, the peak, the plateau of productivity. So, you know, how do we, how do we get there? And, and I don’t, you know, that’s where it feels like we are now. And so I don’t have any, you know, grand concerns about you know, like where it’s going to end up. It’s just, you know, that, I guess the concern might be that, you know, people will give up too early or something like that.

Jeremy Nelson (26:50):
Sure, sure.

Mark Newman (26:51):
And it might, it might just take awhile right? To really get there, but we need to keep at it or we won’t get there.

Jeremy Nelson (26:57):
Yeah. No, I, I agree completely. That’s why I’m here. No, this has been great. I, this has been really fascinating to learn a little bit more about how we all got here. I really appreciate you spending time with us today. I appreciate all the work you’ve done to help get Michigan to this point. So thank you very much.

Mark Newman (27:18):
My pleasure. It’s been great talking with you.

Jeremy Nelson (27:21):
Take care.

Mark Newman (27:22):
Alright. Bye bye.

Jeremy Nelson (27:35):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.