Jeremy Nelson, Director of the XR Initiative

We are excited to launch our new podcast, MiXR Studios, a podcast about the world of Extended Reality (XR) within the University of Michigan and beyond. I am your host, Jeremy Nelson, director of the XR Initiative at U-M. This podcast will explore the work of faculty, students, and staff at U-M that are building, teaching, and researching XR experiences. I couldn’t think of a better guest for the inaugural episode of the podcast than Michael Nebeling, our XR Innovator-In-Residence.

Michael is doing amazing research in extended reality and in accessibility and human computer interaction. We touched on a variety of topics in our 30-minute chat, including his research, the opportunities and constraints of XR technology today. He discussed the difficulty when launching a new research initiative between embracing existing technologies or pushing out toward the future and exploring what might be available five years down the road.

We also chat about how the reality we find ourselves in today, separated from research labs and working remotely, presents challenges, and ways in which extended and augmented reality might be able to provide assistance.

He also discusses why the School of Information is a great place for XR education and learning, with students who embrace the technology from a multidisciplinary perspective with business students seeing entrepreneurial opportunities and medical students seeing how it could revolutionize health care.

Michael also talks about what students and instructors need to understand and embrace XR going forward, particularly resources that help address costs and technical expertise. That is why we launched the XR initiative at U-M and Michael is working with us at the Center for Academic Innovation to develop a MOOC that will address key issues researchers and instructors deal with in the XR space as well as resources and tools available to better understand the technology.

I get his take on what topics we should explore in future episodes, but I’m also curious what you want to hear about and who you think would make an amazing future guest. Email me at [email protected].

Subscribe on Spotify | Subscribe on Apple Podcasts

More About Michael Nebeling

Michael Nebeling is an Assistant Professor in the School of Information, where he leads the Information Interaction Lab. His lab investigates new methods, tools and technologies that enable users to interact with information in more natural and powerful ways, and also make it easier for designers to create more usable and effective user interfaces.

Previously, he was a postdoctoral researcher and lecturer at the HCI Institute at Carnegie Mellon University and the Department of Computer Science at ETH Zurich, where he also obtained his PhD.

Transcript: MiXR Studios, Episode 1

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson and today we are talking with Dr. Michael Nebeling, who is an assistant professor of information at the University of Michigan School of Information and we are going to be talking about all things XR and Michigan coming up next in our MiXR podcast.

Jeremy Nelson (00:31):
Welcome to the mixer podcast inaugural edition today I have with me Michael Nebeling. Hello Michael, how are you? Hey Jeremy, how’s it going? I’m good thanks, how are you? It’s going great, it’s good to talk with you today. I’m excited to have you help us kick off this podcast.

Michael Nebeling (00:49):
That’s cool. I’m excited that we’re doing this. I think it’s a great idea. So what should we talk about?

Jeremy Nelson (00:55):
Yeah, well I’d love to hear a little bit more about all the work you’ve been doing over the last four or five years, how you’re teaching about XR or maybe some examples of interesting uses in education and teaching and learning and yeah. Then we’ll talk about the future.

Michael Nebeling (01:11):
Oh yeah, that’s cool. The future, uh, yeah, we’re living the future setting, right? So some of our students always like after working a little bit with me, they get why I’m actually really into this topic because I can buy all these toys and then make it research and things like that. So yeah, I mean, so I’ve been working on XR, uh, things really focused I think since 2016 so quite a bit now. And it started with me learning myself more and more about the technology. So I created two new courses, and used these as a mechanism to force myself to really think about how do other people teach this kind of uh these topics. And, for example, I took a look around and I have a several friends now, that have been teaching. They have been veterans in this space for a long time and they have usually computer graphics based courses.

Michael Nebeling (02:05):
So I thought, nah, that’s not the way I should teach it here. I’ll teach more, teach more interaction design students, I should approach it differently. So yeah, I need to find my way into how to teach this, and I feel like we have like two courses now. One is prototyping oriented design oriented, where students learn about some of the tools but spend a lot of their time thinking outside the tools. That allows them to be, you know more, in some sense more creative. Cause when I work with a lot of people that actually have worked with a lot of current technologies, they feel they’re so constrained and limiting. And so I feel like thinking outside the box, this is a really cool important skill to have. It’s actually great to work with people that haven’t seen some of the devices yet. Um, because uh, many of them are still very limited.

Michael Nebeling (02:54):
So we teach those courses and one is more technology oriented, more like programming. It’s, you need a bit more of a tech background. And then I do research in my lab, we are in the human computer interaction research space. So we really look more at the intersections and the interesting issues for users of these technologies, and those can be end users and designers. And so how do we empower more designers has been a big theme. Yeah, I feel like I’m doing that. And then I’ve been working with you now for, I guess we’re still in our first year, but you know, taking this to scale at Michigan, I think that’s really cool.

Jeremy Nelson (03:35):
Yeah, no, it’s super exciting. I’m so excited to have you be part of the group and helping us lead that, and I want to jump back to what you just mentioned. So I think it’s a really interesting way to approach this space from a human computer interaction angle and design and prototype. Cause I don’t see a lot of that out there, and I know we’re still kind of early days in certain parts of this of what are good interactions, what are good designs. So what are you finding in terms of what the students have been able to do, what your research has been able to bring to the space from this perspective?

Michael Nebeling (04:12):
Yeah, so it is interesting. And so one big theme that I find is, um, is this legacy oriented approach where we’re trying to figure out whether the things we are used to still work in XR. So do these gestures still make sense when you don’t do them on your touch phone, but you know, do them mid air. Uh, so there’s a lot of this exploration and there’s, this um, I think, right now whenever we do a new research project it’s a very interesting decision, do we want this to be available and ready right now. So for example, the classroom, should I design around smartphone based AR interactions because not every student has a Hololens for example yet, or should I be thinking forward and think of the HoloLens as a current prototype of a device that might be more adopted. Uh, like this type of wearable AR headset in five years from now when everybody has in the classroom.

Michael Nebeling (05:05):
And you can, you know, the interactions that you envision then are very, very different. So I feel like we often approach these kinds of, uh, well this thinking about this technology and like also understanding that why we’re studying these technologies and the impact on users. Also how they are evolving very, very quickly and rapidly, so it’s very interesting like, how far into the future do you want to go in your research? Um, so clearly you don’t want it to be outdated too quickly. Um, the human computer interaction side is very interesting because, uh, the adoption in itself is still an issue. So obviously there are lots of reasons for that. It’s still very expensive, but really also at the core, the designs, they’re still very clunky and cumbersome and we’re still exploring even all these, we would say professional products are still in the early stages. Everything still feels early stages. So it’s a really cool space to be doing research and design, so lots of students are excited about this. And, uh, I, myself, I can see myself doing this for quite a bit longer. It’s really cool stuff that we still need to figure out.

Jeremy Nelson (06:15):
Yeah, no, I think, and it’s, it’s, it’s a really, I think, unique perspective, you know, maybe talk a little bit more about, uh, you know, the user experience design, background from Michigan and, students coming from that space, but into this 3-D space or kind of these, what are they bringing to that space? Or what are some of the challenges you see?

Michael Nebeling (06:37):
So I, I’m actually housed in the school of information. And so I work with a lot of, uh, interdisciplinary students or students with all kinds of backgrounds are taking my courses, not just programmers. And so they bring in very different angles. So the business student sees all the entrepreneurship opportunities for example, the medicine student sees how it could revolutionize a lot of the, both the training but then also really procedures and actually really may help us do a better job in medicine. So how people approach this topic is, um, is very different depending on the, on their background. Um, now the user experience, I would say that, uh, first thing we really have been trying to do is to figure out which of the things we have established. So user interface design has been evolving. People often use the iPhone and the iPad as this example that has revolutionized and transformed mobile and smartphone.

Michael Nebeling (07:35):
And smartphone is hard. Just, you know, it’s hard to imagine a day without, I mean in many areas of the world, let’s say a day without a smartphone. But like if I took your, your quest away, do you think you will notice what kinds of tasks can you, you know, you’re not really relying on that technology yet, but I think this will come and, um, yeah, and then there are also all the dark sides I think, which we are also exploring. Um, obviously in currently the world is exploring most of it through the movie industry, but also in research. Uh, we think a lot about privacy, security, um, safety of users. Um, and obviously the vendors of these devices are also putting in some effort there.

Jeremy Nelson (08:23):
Yeah. I mean it’s always that conflict between, you know, academia and business and where the constraints are or the vision to move it forward.

Michael Nebeling (08:34):
Yeah. Conflict is a good conflict is a good term here. A good word. Uh, I mean, yeah, you’re right, there is, let’s say there is a gap, um, because researchers like to think forward in terms of five years, but, and we have solved this and that, that it actually in practice we haven’t. And so our research that takes a specific angle, it’s like, okay, you know, privacy is not my research area so I can just ignore this for now. And I focus really on the cool and fancy interactions. So that’s why also to be honest, like a lot of the research, because I can choose to focus on topics, zoom in, zoom out, or actually leave some, as secondary or future work for others. As a company, you have, you have the responsibility to deal with all these issues at the same time, which is why maybe progress overall seems slower. And what we see, like when you put on a HoloLens today, let’s say the Hololens 2 or something like that, um, this is not where we are on research. It’s still like five years, five years behind research in some sense. But for many of us it will be their first real experience. Um, and so we think that is the state of the art. But actually in research, it’s not. Lots of clumsy, interesting prototypes in research, so.

Jeremy Nelson (09:47):
Yeah well, I want to go back, you know, you started to mention students coming from cross-discipline and you know, here at Michigan we have 19 schools and colleges, a wide array of programs and disciplines to offer education. What are some of the best examples you’ve seen of AR VR in the education realm?

Michael Nebeling (10:09):
I think we are beginning to see some cool examples. Um, so obviously, you know, this is the thing that professors always like to talk about their own stuff. Uh, so we have worked on, we have worked on and I think two projects that I was very excited about. One was in the accessibility realm, so it was called iGym. It’s with a colleague Roland Graf here who you should probably talk to. Um, he’s in the school of art and design. And um, that one was a project where we used like a, basically a top down projection onto like an actual floor that turns that floor into an air hockey field. And then children with or without, uh, wheelchairs can, can do like an adaptive sports session and play against each other even competitively. And so the AR, the AR component of that system was from an AR a design perspective.

Michael Nebeling (11:00):
Um, fairly straightforward. I mean you had to simulate good physics and have that virtual ball fly around and respond to the participants. So we had to track the children and wheel chairs and the interactions. Um, but I think the larger, the larger scope around this project was like this leveling the playing field. And how do we design this? How do you balance it? Because we actually over frankly overbalanced it so that we gave more power to the wheelchair users so that they could, you know, more or less accidentally easily beat, uh, non wheelchair users. And we played, we started, you know, we tried this out with children and it had interesting effects like for the first time when you then beat the other player and you haven’t had that feeling a lot before. It’s very interesting how you respond to that. The other example um is let’s say in the classroom.

Michael Nebeling (11:47):
So we have a course here Ed Happ and then, um, Michelle Aebersold. She is in nursing and Ed is in the school of information and so they together, um, so Ed’s course is really on crisis simulation and they’re really trying to prep students for like these what is, when your plan B doesn’t work until you have a plan C scenario and there’s no communication by the way, and you don’t have any water and you have to figure out a way to make it out of here. Um, and so we took this and they improvise a lot in current learning activities that transformed the classroom, like literally into a, um, war, well into a war zone, playing, Saving Private Ryan and this kind of stuff and um, to impose some pressure on the students. And we, we came in and created the multi-user HoloLens experience with injuries and like an escape room scenario.

Michael Nebeling (12:39):
And I think in many ways, you know, it was really interesting to design all this and implement it and get it to work and work around the networking issues and the tracking issues. And well after all that, I was still a little skeptical. That’d be really transformed the learning experience. Now I think we created a really cool example that we can learn from, but it’s not like, Oh, now we can deploy this everywhere. And it’s like so easy. So it raised more questions than providing answers. But I really think positively about these projects and I think a lot of the examples we see right now are in that space. Everybody’s experimenting a little bit, right? Then you, if you’re more an entrepreneur, you’re, maybe you’ll, you, you see more hype around some of this. If you are a researcher, you are more critical self critical about that. And I think this balance, um, together with industry, there’s no way without industry, but with industry we can figure this out.

Jeremy Nelson (13:34):
Yeah, no, I think, you know, we’re all, we’re, we’re thinking about that from the XR initiative standpoint of how do we scale out and deploy and keep content fresh on multiple devices. And it’s, yeah, it’s a big challenge. It hasn’t been figured out and hasn’t been matured to the same level as smartphones or you know, virtualization from desktops and PCs and pushing out content that way. So there’s still, there’s still a bit ways to go. I mean we’re optimistic about implementing some of our projects, uh, in this space and seeing how we help solve those problems. I mean, kind of in that realm maybe, and you started talking about the dark side earlier, I mean, do you have concerns about the future of XR for teaching and learning or what are some of those risks that we should be paying attention to?

Michael Nebeling (14:22):
I think it is, it is very interesting, especially in the current time with Covid-19 happening and a lot of us teaching remotely the way the students, some students embrace the technology. Like we have these virtual classrooms now on like online video, livestreams. And then it is interesting how many students, for example, choose to show the video and the concerns they have about that. And um, I have never been able to experience it at that scale as we do it now. And the variety of concerns that students have, like will other people be judging me about the things they can see in my video feed? And what is it with other, you know, I live with people, for example. I mean, um, what if they walk into my video feed and things like that. And so, um, it is interesting because there’s a lot of parallel to these AR/VR technologies because they all have built in cameras.

Michael Nebeling (15:20):
They all look at the world to, well, to help the device see the world, but you know, at the same time they do obviously also then see, um, passersby and all kinds of people around us. So yeah, that’s um, I think it’s, um, I think it will be very hard for us to figure this out, um, in a way so that it works for everyone. Um, I also like as an instructor, uh, uh, I want to increasingly make use of these technologies in my own teaching, but I always also have to think about opt out and backup plans because different students react differently to virtual reality and motion sickness. This is a thing, for example and so yeah, so it’s interesting.

Jeremy Nelson (16:07):
Yeah. Well, and I don’t even know if we should be thinking about it for this is a solution for everybody for everything, right? This is another technology, another way to experience content or experience pedagogy that we need to find where those places are, where it makes sense and where it actually helps the experience. There’s a lot of discussion right now about in this remote space, like how do we solve the problem around studios and labs and these, you know, physically immersive group experiences when we’re remote and we’re looking through a screen.

Michael Nebeling (16:43):
Yeah, I agree. And I mean from an HCI perspective, if you’ve learned anything in HCI, that it’s like every user is different and there is no average one size fits all kind of user or one size fits all average use a one size fits all solution. I think what is happening though in the XR space, because it is still early stages and it’s very difficult to create these experiences, we don’t actually have a lot of support for customization or personalization in most of these apps. They’re designed for a one size fits all, they assume a standard height of 1.6 meters and uh, you know, left handed versus right handedness is something that I don’t know, I feel like, um, well one thing that I’m for example right now particularly interested in is like if I’m going to work on a new project hopefully soon where we want to use some of these devices, um, with visually impaired users.

Michael Nebeling (17:37):
And now if you think about, for example, where the cameras are placed and what these AR devices expect in terms of how users navigate to help them see the world, because at any given time a camera cannot actually see everything around you. They don’t have 360 degrees of view, they have a certain angle as a visually, um, directed user, let’s say, uh, or oriented user, you really look at where you go. But if you have visual impairments, you may not and navigate the same way, and so you may not actually feed the necessary information to the AR, um, uh, headsets. So it may not be able to give you the information that could really help you navigate safely in the environment, and then we may need to rethink how we design some of these devices. So, and, and that suggests that we probably need more flexible devices in the future. Now, whether that means that each and every device needs to be adaptive or adaptable or whether we just have different types of devices, that the latter scenario is more, more likely that there will be, uh, different classes. Just like the smartphones and all kinds of technologies.

Jeremy Nelson (18:41):
Yeah. Yeah. I mean, it’s follows probably the other technological advances in similar spaces. Um, well, you know, we’re, we’re really trying to bring XR for teaching and learning broadly across the University of Michigan. I mean, what do you want to see Michigan do to help advance education and what it means to get a Michigan degree?

Michael Nebeling (19:03):
Well, I think the first really important thing for us was to get this XR initiative going. I really liked, uh, the emphasis of the initiative, this idea of really helping, um, obviously in some sense students to experience these technologies in the classroom, but also really facilitating and helping instructors because the main issue that we still have is that there seems to be such a high cost to thinking about using this technology and embedding it into your classroom. So if you have zero assistance, um, then it’s really difficult for you to really, I mean, I’m somebody who likes to play around with this, but even I sometimes complain about all this stuff I have to take to a course and maintain. It’s this device charged and that device charged and why this demo was just working yesterday, why is it not working now? And I think the XR initiative in many ways will facilitate this plus obviously also thinking a lot about the pedagogy where I think we still have a lot to learn and the only way to learn is by doing so.

Michael Nebeling (20:06):
Yeah. In terms of specific things I would like to see at Michigan? Yeah, I think we need to create more spaces and opportunities for the students to access for the students to access these technologies and for instructors to experiment and try them out. I think the ideas of workshops that you have and like all the work that you’re doing to the, for example, the beginnings of these speaker series and uh, this working groups around different topics with faculty that you’re organizing. For example, around remote instruction. I think those are really important efforts would like to see more of those together with obviously our friends and colleagues who need to decide on the good topics for this. But I really like it something regular where we exchange our experiences, learn from each other. This podcast I think is another great example of what I wished we could do because it allows me to learn about a lot of, I mean, you can tell me what it looks like for you to come here. You knew there was stuff going on, but didn’t you spend quite a bit of time figuring out who’s doing what here?

Jeremy Nelson (21:18):
Yeah, no, I mean I probably spent a good two to three months just trying to meet with as many folks on campus, you know, doing something in XR, interested in XR, you know, had no idea what XR was but you know, was curious and so yeah, I mean it’s a, I’ve talked to over a hundred people, um, just in the university. So there’s a lot going on, there’s a lot to bring together and to learn from. And so that’s one of my goals with this podcast is to really, uh, educate, you know, the university community about what’s possible, what’s going on, what are people doing and how do we all share in that and grow from there.

Michael Nebeling (21:55):
Yeah, me personally, I’m very excited still. I’m ok with making mistakes on the technology side. I’m afraid that if we design everything, if we try to be perfect all the time, we sometimes constrain ourselves maybe in some of the innovation or to be honest, some of my, some of the coolest features I have developed in some of the systems that I’ve created in the past. They were like, by accident. And so I’ll and as long as it doesn’t obviously cause harm, I think it’s good. Uh, and so I feel like what I also like is the, the different levels and the different talents we have across Michigan. So there are people that can focus more on some of the technological issues. Then there are people that are really focused on the pedagogy and each in each group we still have a lot to learn cause XR is transforming each of these spaces and then through the XR initiative and a lot of your efforts we can work together and this working together, this is where a lot of the magic will happen over the next few years I think.

Jeremy Nelson (23:05):
That’s great. I agree wholeheartedly. Well obviously this is our first podcast. What other topics should we explore as we move forward?

Michael Nebeling (23:14):
Yeah, it’s great. I think the, the focus around the education and um, classroom in the broader sense. So both like, uh, local, co collocated and local remote MOOC. Uh, all these kinds of, I think that’s a really cool thread to keep throughout the podcast because this is how we can all connect to each other. You will talk to I guess more in, um, you know, professors, instructors and also maybe some of our students. I do think, um, it would be nice to hear a bit about our student efforts. So the alternate reality initiative from some of our students. I think it’s cool because it could be, it could set a good example for other universities. And then even within Michigan, like, uh, the more faculty learn about what our students are doing, the more they realize all the potential and the enthusiasm around these technologies and maybe it’ll make them feel more comfortable to experiment.

Michael Nebeling (24:08):
I think, um, I, you know, in teaching evaluations, sometimes, I got this kind of like Michael wasn’t well prepared for lecture, but I think what students were actually referring to was the issues that I had with some of the technologies. And I’m okay. I’m okay with being dinged for that a little bit because I think in the end I’ve learned something and I wouldn’t, and I know next time I need to bring two cameras. If one camera is used for AR marker tracking, I can’t use it to record something else. So you know, things you don’t consider or don’t always consider beforehand.

Jeremy Nelson (24:39):
Yeah, well that’s, and that’s the real world. I mean, so seeing how to solve those problems and work through it, it doesn’t work perfectly all the time. I do want to jump back. You mentioned MOOC as an area of growth and you are producing a MOOC right now with the Center for Academic Innovation here at Michigan. I’d love to hear a little bit more about that and what your goals are and where that’s going to go.

Michael Nebeling (24:59):
I think it is another important way in which we can reach more people, to understand and learn about XR technologies, both the potential and how to actually really get started working with them and then also get an overview of all the issues we should concern ourselves with. I mean you’re potentially designing virtual environments that could cause harm to people. Like just last week I had a student, uh, she’s been working with me for a year around, uh, a museum of memory. And what is a good use of AR when it comes, when you could potentially accidentally, um, revictimize people. But you want to use AR, uh, as a means to, you know, for empathy and like, you know, bring some of these topics kind of like back into memories. It’s very, it’s very interesting. So the MOOC is focused on, I think of it as like, as I would like to think of it as this really collection of resources that not only I myself can use among classes, but also other, um, professors here at Michigan could use, um, you know, to point the students to, ah, here’s how you learn about this is current issues are, this is, um, this is where we have done, where we have made some good progress.

Michael Nebeling (26:20):
So education is going to be a little bit of a theme in some of the things of that MOOC, but really more like from the perspective of learning and teaching, a big goal of my MOOC that I give tools to other instructors so that they can actually embed, even if they don’t teach AR/VR courses, but let’s say they teach an interaction design course or you work at the library and you have students come in for some kind of prototyping activity. And so there could be a video and a set of exercises and maybe even a quiz that could be useful and you could design an activity based on some of the resources that are provided as part of the MOOC. So it’s not just like this, here’s a course, and that’s one way to approach this MOOC is actually a set of three courses. One is focused on learning about the concepts and shaping the future, taking the critical issues into account and learning about those issues, which is very important.

Michael Nebeling (27:09):
And then the second one is really focused on design. So what can you do if you’re not a hardcore programmer? Is there a space for you in that space? And I think there is, even though most companies still hire in the technical space, the third one is then really on developing with current technologies, um, has its own issues as I said, right. It kind of like the more you know about the limitations of these technologies, the more you think within these limitations. And so, I’m not sure how much this has happened to your team yet, but if a faculty member is so excited, I want to transform my classroom experience, can I do this with HoloLens? And you’re like, Nope, “we haven’t said it quite like that, but yes, I understand”. Yeah. So I think that’s why the set of courses is very important. We can cater to a broad audience and everybody can go as deep as they want.

Michael Nebeling (27:57):
And I really hope that that people will look at it as a resource and share it and use it in interesting ways and let us know how we can improve it. Um, I’ve been working on this and you’ve been in many of the meetings. Uh, as you know, I’ve been working on this for a long time and I’ve, I think I’m in the fifth, the fifth way of how I could make this more accessible. Um, and also teaching it at scale. I may have been overthinking it, I should have, I should have pushed myself, but I really want to get this right. So I hope it’s going to turn out as a nice experience for our online learners and maybe colleagues even here at Michigan.

Jeremy Nelson (28:32):
I mean, I think it’s going to be great. I’ve seen some of the behind the scenes and I think it’s really going to help folks interested in this space and want to know where to get started and how to get moving and use that in their career or in their learning and teaching space. So this has been a great first podcast, Michael, I couldn’t have hoped for a better first guest. Thank you so much for all of your insight and thought in your work. I really appreciate it.

Michael Nebeling (28:57):
Yeah, thanks a lot for having me. I, so should I nominate somebody now or how do you want to do it? I think there’s lot of cool people you will meet throughout this podcast. I’m excited. I will follow this. Um, it’s a good way to keep up to date and learn new perspectives. I’m glad you’re doing this, so thanks for having me.

Jeremy Nelson (29:16):
Thank you. Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at

Jeremy Nelson (29:34):
ai.umich.edu/xr