Three-course MOOC specialization focuses on the Knowing, Doing, and Shaping the future of extended reality

Jeremy Nelson, Director of the XR Initiative

In this week’s MiXR Studios podcast, we talk with Michael Nebeling, assistant professor at the University of Michigan’s School of Information. Michael is also the XR Innovator in Residence for the XR Initiative. We conducted a podcast recording in front of a live audience where we discuss Michael’s new “XR for Everybody” specialization set debuting Nov. 9 on the Coursera online learning platform. We also discuss the first year of the XR Initiative. We covered a number of topics from Michael’s history of teaching AR and VR design to the incredible amount of work that he and the team at the Center for Academic Innovation put into the creation of this new MOOC specialization. 

The XR for Everybody specialization is intended to be accessible to anyone that is interested in XR technologies and wants to learn more about what they are, how it can be done, and what the future of XR will look like. There are three courses within the MOOC specialization and they are focused around the Knowing, Doing, and Shaping the future of XR. Michael describes his intention for creating a course like this and how this relates to the way he teaches XR design in his residential courses. He takes us behind the scenes to learn about what the production was like and how everyone had to adjust mid-production and create the remaining content remotely because of the COVID-19 pandemic. 

In XR for Everybody, Michael brings a new level of immersion and production to engage learners on what the technology can do today and where there are constraints. The learners see Michael teaching in VR and using AR devices such as the HoloLens to map out spaces to understand object identification and tracking. The specialization is more than Michael lecturing and showing off cool technology, and learners will use tools such as Unity and WebXR to create their own experiences and peer review other learners’ projects. 

In addition to all of the time Michael put into teaching his residential courses, doing research, and creating the XR for Everybody MOOC specialization, he spent a lot of time with the XR Initiative helping shape its first year. We discuss some of the accomplishments from this first year, in particular the XR Innovation Fund, which awarded eight faculty projects from seven schools and colleges at U-M. Michael was instrumental in helping review those proposals and assisting in the interviewing and support of our two XR Developers, Eric Schreffler and Moeezo Saleem. I am appreciative of all the work that Michael does and all the support he has provided to me and the XR Initiative. Thank You Michael!

We conclude our conversation with an open Q&A with the audience to explore topics around how students can engage in more design, what ethical considerations are there with this technology, and should you build your own solutions. We had a great audience and with one of the participants you could really see how passionate Michael is about his work and how he teaches in a way to encourage his students to think broader and explore the possibilities. We want to thank all of our attendees for joining us and participating in the discussion.

This was a fun discussion with Michael and we look forward to participating in his XR for Everybody MOOC and seeing how the learners engage. Look for our notifications about the launch of the XR for Everybody MOOC on Coursera. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 21

Jeremy Nelson (00:00:10):
Hello, I’m Jeremy Nelson. Today. We have a very special episode from a live podcast recording with dr. Michael needling, who is an assistant professor of information at the university of Michigan school of information. We are discussing his new XR for everybody MOOCs and year one of the XR initiative coming up next in our mixer podcast. Well, cool. We’re a couple of minutes past there’s folks can join in. Why don’t we, why don’t we get started? Uh, so thank you everybody for joining us today for a, a live podcast of mixer studios. I’m excited to be joined by professor Michael nibbling. Welcome Michael.

Michael Nebeling (00:01:01):
Hello, Jeremy. How’s it going? My friend.

Jeremy Nelson (00:01:04):
It’s good. It’s good. I am looking forward to our conversation today to talk about your XR for everybody MOOC and just share some of your goals with that and maybe, you know, your process behind it, why you thought this was important. Um, and then maybe spend some time talking about your role as the XR innovator in residence this first year with the XR initiative, we just celebrated our first year and we’ll be talking about some of the highlights there, and then we’ll open it up to the audience for Q and a, and it folks want to put questions in the chat or the Q and a, and we’ll we’ll address those, uh, after we’re done. Sound good.

Michael Nebeling (00:01:42):
Sounds great.

Jeremy Nelson (00:01:44):
Great. Well, let’s, let’s talk about the, the MOOC was in process before I started before the XR initiative. I think you had just kind of signed off or were in the process of kicking that off. How did this, how did this come about? I know there was a teach-out a couple of years ago, and I know this is your area of expertise and I’d love to learn a little bit more about that.

Michael Nebeling (00:02:02):
Okay. So the first thing I’ll say is, um, so quite a few people involved and the support from academic innovation has been instrumental. And I actually blame mostly, um, James DeVaney for taking me out to a nice coffee and, um, saying Michael, you know what, we should be good to do this MOOC specialization. I think this could be, I mean, it was very encouraging. So yeah. So before that, you know, when I joined Michigan and 2016 young, Michael, how do I do as a professor given all that money, let’s buy some things. So we started on the AR/VR space. I was really fascinated about these as interactive technologies. And I had to learn a lot of this stuff myself. I think that was an important first thing to realize. Um, a lot of my I’ve worked with a lot of these technologies, like the Kinect, for example, which can see your and track you skeleton and gesture and interactions.

Michael Nebeling (00:03:02):
I’ve done voice-based interaction before. Didn’t find it so interesting from an interaction design perspective. And so the AI technologies were really appealing to me and they were just like being, I mean, that’s been around for many years, there are a lot of people in the, in the research space that would normally discourage you. Like, why would you start in that space doing research for 30 years? Who are you? And so, yeah, so there was some interesting obstacles, I think, along the way I’m, I’m, I’m, I’m not, I’m not like, like not one of those gods VR/AR gods, but, um, we’ve been like with my little team we’ve been quite successful. I think in the research space, I’m proud of what we have accomplished. We definitely had challenges there and this realization that, you know, none of this stuff myself, like, I think I’m not like stupid or anything.

Michael Nebeling (00:03:51):
So I should, um, really felt, Hey, there should be an approach to this that is not the computer graphics route like that doesn’t require that, you know, all the three D programming and that she likes super familiar with unity. And I was actually afraid of unity for a long time. My background was in web and mobile. So how do I translate those skills? So I started to create two courses. I was teaching interaction design for a few years at Michigan and everybody was doing mobile apps. So I was getting a little bored and saw an opportunity to really do AR/VR. So I started that interaction into an existing interactive design courses, but then I felt like I need to provide more training and more context. And so I had lot of support from the teams to create my own courses. I think it’s a, it’s a really fortunate thing to, to, to be able to teach none of these.

Michael Nebeling (00:04:41):
You have to teach courses foundational ones, but to create your own ones. So yeah, school of information was really supportive, Tom, Finholt. Um, and Beth Yakel have been like pushing me also like in a positive way anyway, so we have these two courses now. So like I’m jumping forward. We have like intro courses. One is a interaction design, for AR/VR teaching right now, two sections, we are about 80 students, really exciting as this undergrad masters that were very masters. Uh, there might be some might be some undergraduates. I usually am inclusive. Um, but we also have like a capacity limit and the COVID-19 challenges. So anyway, lots of things going on there. Anyway, residential courses, we have a more technical one in the winter. Um, so in January, starting in January, I’m a little scared of that one. It really requires access to devices.

Michael Nebeling (00:05:36):
And now we are a little limited there. And so the online course was like, okay, we have to go online. Anyway, like at the time, I didn’t know that it was mostly like James academic innovation, very efficient was a somebody I’ve consulted with a little bit. He told me about the culture of academic innovation, how cool you guys are, then you came on board. So I was like, okay, cool. Um, the, all the stars are like in the right position, let’s do this MOOC. Sure. And so, yeah, we, we worked together on this MOOC for a year or three MOOCs. I mean, I think that it would be good to know a little bit about the complexity. I can go into detail there. I want you to structure the discussion by like quite a complex thing to actually create these three MOOCs. There’s an interesting interplay between what happens in the residential course and in the online course and the vision I have for that, that I haven’t really fully implemented yet can talk about that as well.

Jeremy Nelson (00:06:29):
And so who, who is the intended audience in your mind, who, you know, obviously you’re teaching courses for masters students at the university of Michigan on campus, when you thought about the MOC, like who did you want to address? What problem are you trying to address in that space?

Michael Nebeling (00:06:46):
Yes. So the Teach-Out that you had mentioned, which I haven’t really picked up on yet. Um, so that was a bringing this understanding of, Hey, there are these XR technologies that are coming into our everyday lives. We need to think about the issues. So it was really a raising awareness, understanding what AR what VR is a very broad audience. So then we got a little bit of feedback. It’s a nice introduction to AVR, but how do we design for it? How do you develop with these technologies? So that was missing from the Teach-Out. And I did this in my residential courses. So I think of the specialization that we have now is like one course is focused on this broad understanding, both conceptually and technically, and that is targeted to like everybody, even those that would never want to actually be a designer or developer who might be users, whether they know it or not.

Michael Nebeling (00:07:35):
Right. Because you could be a, a passersby like, you could just walk through an AR experience and being recorded by another device and, and being afraid of what that technical. But if you know, that doesn’t know anything at this stage, it’s just much more comfortable how you interact with these technologies. And the other two courses are really going deep in design and development, and they are targeted towards people who, who want to learn more about potentially transfer some existing schools, but maybe also the first time they’re doing interaction design. So it’s like, let’s say the courses are not super advanced, but they’re really providing hope and I wish a good foundation.

Jeremy Nelson (00:08:13):
Great. Great. And you’re, and you’re coming at it from a different angle of user experience design or interaction design versus what’s I’ve seen traditionally as coming from the game video game design development space. Was that, was that on purpose or that’s just the space you come from. And so that wasn’t even a consideration to come from the game dev side.

Michael Nebeling (00:08:34):
So I commented as a human computer interaction researcher and interaction design. As a course, I’ve been teaching years, I’m just teaching these very, um, very traditional now web and mobile. They are actually traditional. Now. They were exciting at the time, like thinking about touch and gesture. Woo, exciting, uh, was cool at the time. But I think the next wave is AR/VR into these technologies. And so right now I think I have specialized courses, right? People still think of these technologies as being exotic, like AR/VR. I know I’ve seen that. Um, and, but I think what is happening in a few years from now, a lot of what we now have in these, in these MOOC courses and what I teach residentially, um, that’s how user experience, design and interactions are going to be transformed over the next few years. It’s already happening. Web web design. You don’t, you don’t think about desktop anymore. You think about desktop, mobile smartwatch, maybe AR/VR, and, um, that realization I think so it’s interesting to think about how that MOOC itself, will evolve over the next, the thinking around it believable.

Jeremy Nelson (00:09:46):
Sure. And then this will be open to anyone, right? And anybody in the world could sign up?

Michael Nebeling (00:09:54):
So the way we have designed it, it, uh, when I say accessible, so I have to be careful. So accessibility in this space is really a super big issue. So I say the course is accessible to a lot of different people, and it’s a series of three courses. One is really providing the foundation. So what if you’re teaching, if you’re taking a course in residential program that course would have zero prerequisites, no assumptions whatsoever. Everybody can take it. Everybody should benefit. Everybody will grow throughout this course. So you’ll know more when you come out then potentially at least in this space, then when you come in, um, the design one in the middle is still a seminar, no prerequisites, like my, my residential question, I’m teaching, uh, no prerequisites. You just come in. And probably some, I throw around some interaction design terms, like domain context constraints, affordances, uh, that sometimes students remind me that, um, we haven’t heard of those yet.

Michael Nebeling (00:10:53):
And yeah. So yeah, step back and explain it. Um, and then the one where you really play with these technologies, and I’m sure a lot of our, um, attendees here would agree with it. There is still a very high barrier to, um, working with these technologies directly. And that last course is trying to break that barrier. Like you won’t, you will probably not be like you will be in a position where you can implement your app ideas, prototype them, but playing the, working directly with the technologies, not being limited to some prototyping tool, but working directly with Unity or A-frame. And that is a great accomplishment, but that re that is a higher barrier to entry already, but we’re breaking that nice. And then I think there is a ceiling to the course, which is like, if you’re like, if your dream is to work on the Microsoft HoloLens team or the Facebook Reality Labs, thinking about Oculus and what the, the concerns that industry at the moment are obviously content is one thing, which I think we are empowering more people to participate in design.

Michael Nebeling (00:12:00):
But the issue is that a lot of industry works on are much lower level, hand tracking. Uh, how do we detect objects? How do you interact with the environment? How do we make it safe? So, um, I think that will be the next step after the MOOCs. It’s not, that’s a little out of scope, but I talk, I have a lecture on advanced techniques, I talk procedure, procedure, regeneration, 3-D reconstruction, fancy things, but not, not, uh,

Jeremy Nelson (00:12:30):
Those and those aren’t part of the Honor’s track. Correct. What is the honors track?

Michael Nebeling (00:12:35):
Yeah, so that’s a good point. So we designed the courses to be, you can, you can learn about XR while you’re exercising, or like dreaming about something else, just listening, watching some cool demos or, or you can get your hands dirty. And the Honoro’s track, uh, is an ex, um, it’s an experiment in some sense, um, to get, to get this, to work with a lot of different devices.

Michael Nebeling (00:13:04):
It’s one of the main challenges we had. We took the lowest common denominator. Like for example, I feature the HoloLens 2 like, but my course is not a HoloLens 2 course, right? Maybe Microsoft would have one focused on how those $3500 device that, um, could be barrier as well. So sorry for he. So I don’t exactly know how Coursera works and how, uh, paying or whatever for that works, but all the content is available for free. And then if you want to get a certificate out of this, there might be some costs involved, but you can still access all the content. And the Honor’s track is designed for people who really want to spend some time with these technologies who want to add a project to their portfolio who want to be critiqued by other learners. There’s a peer review component. Um, and basically over the, over the course, the second and the third course have this, um, project oriented Honor’s track.

Michael Nebeling (00:14:13):
Um, you actually develop to what I think will be significant portfolio pieces. I really hope that will be online learners. Cause like, Hey Michael, I did this in your course and I’ll be blown away. I’m sure. And, um, the first course also has an Honor’s track, but that one is, Hey, let’s discuss these issues of accessibility, of equity, of social and, um, privacy and security concerns, ethical concerns, right. Um, which by the way, I’ll also in each of, uh, won’t stop talking because so excited. That’s fine. That’s why you’re here. They, they, these, these concerns are also in each of the MOOCs, but when you actually play around with these technologies, like you’re really deep down solving, implementing this app, right? It’s like there’s, this, there’s, this, this library, you can just use it. That’s the job for you that hosts all the data with this evil, evil provider. Then you’re thinking you have to make some really interesting decisions there. And then you sometimes sacrifice maybe like, um, you make the fun, it gives you the features? That’s what you need. Right. And you will sacrifice, right. Privacy security and some other ethical concern.

Jeremy Nelson (00:15:25):
Nothing is free. Nothing is really free. What, in terms of equity, you started to mention, uh, designing for the lowest common denominator. So obviously not the HoloLens, probably not even a Quest or something like this. It’s it’s, it’s, it’s in reach, right. It’s $300. It’s a little more unreached than $3,500. But yeah,

Michael Nebeling (00:15:49):
I’m not sure about the most students would agree that a $400 device is in reach. I was very disappointed when one sec, when this is a message to you guys out there at Google, right? So when you, this is daydream and it’s a nice cardboard, like it gives you a little bit of a controller here, three degrees of freedom controller, one hand only, but still that’s still cool, but it requires a phone to be put inside for this to work. Hey, I can buy this for like $40. That’s the difference? I can do a lot. I can reuse my phone. Like I feel like from an equity standpoint, this was really, really cool. Why are we crushing those products? So now I’m in a scenario where my lowest common denominator approach approach is like, that’s good cardboard. And, um, but

Jeremy Nelson (00:16:40):
What’s cardboard for people that don’t know?

Michael Nebeling (00:16:43):
Is a cheaper version of this like $7. And it’s literally cardboard. And I don’t actually have one here right now, somewhere you’d have to be, um, your phone goes inside and you’ll hold it in front of your head. And that gives you a kind of a virtual reality experience. And it’s super cheap. Do you have one around,

Jeremy Nelson (00:17:03):
I have a Nintendo, I have a Nintendo switch one. Okay. So putting a switch on your head is even more uncomfortable. Cause it’s very tough.

Michael Nebeling (00:17:11):
The one that I like is a $15, but I’m not sure whether they’re doing product advertisements, but anyway, it says I Am Cardboard. Oops. I don’t know which way to hold it anyway. Um, and so then you take it out and it becomes your goggles and the phone goes inside here. And then it gives you a little bit of VR and, and, um, but even at $15, like when, when you teach students at Michigan or anywhere else in the world, there’s already a lot of costs involved in just getting into these programs. Financial burden is very high already. And so who am I to say, Hey, you’re going to take my course. It’s just, buy a $400 headset, you’ll be fine. Right? Difficult. Very difficult.

Jeremy Nelson (00:17:54):
Yeah. And I mean, we’re starting to see prices come down as more people get into this space, but even at the most recent quest, right. I would say they’re $299, but they’re subsidizing the price. I would say for the access to the data, the data that you would generate on that,

Michael Nebeling (00:18:14):
You know, the issue that we have is scale. Like we want to teach this to a lot of people, potentially thousands of learners online, and from all kinds of backgrounds all over the world, we have all kinds of issues already, connectivity wise, bandwidth wise, um, there might be severe implications. I’m always concerned when people started recording, like working on an assignment and the AR assignment and you see cats and dogs walking, that’s cute, but I’m like, I don’t know I’m going somewhere else right now, but let me, let me come back to the, to the equity thing. Um, it’s a, it’s an unsolved issue. I don’t think that, I mean, it’s nice that the quest is there at $299 model. It’s still not, it’s not good enough for most learners. And in terms of affordability, I think we need to be, I mean, there’s a trend at least.

Michael Nebeling (00:19:01):
So maybe it will be in everybody’s pocket in two to five years from now. Maybe I also like this direction of actually working around the smartphone. I really like that we have this tool. We can turn it into a VR thing. We can turn it into an AR thing. It can be a head, a head worn AR with different extensions to it. I was betting on it. I wrote two unsuccessful NSF proposals because my reviewers were like, do you think this is really the way to go? And I feel like, yes, it should be. But, um, it’s not, uh, I can see that now Google and others have killed that idea of cardboard and daydream and Oculus Go.

Jeremy Nelson (00:19:42):
And what outside of financials, you know, revenue generation were there technical limitations with, you know, some of what I heard was, you know, the, the refresh rate or the mobile phone processor couldn’t necessarily keep up and process the experiences. I mean, is that true from your perspective or what you’ve done?

Michael Nebeling (00:20:03):
Well, um, I don’t know how many times you have overheated your phone, but, uh, um, I, I can tell you that it’s a very interesting process when a, uh, phone is inside one of these headsets and the, the air is not really good. And you’re like, you’re running this at 60 frames per second or 90, which is kind of like our 80, I think I’m not sure what they’re rendering it. You can see that when it enters the VR, it, the display starts to look different, not, not just the rendering, but also like the refresh rate. And so that basically squeezing the most performance that you can get out of these phones, they get very hot and then they also throttle. So that means the performance goes down. And so I can see, I see that that is an inherent limitation. So I think the industry wanted to push forward these, um, they have now all of these technologies, whether they can build the cameras inside the headset, the headset, is stand alone, they can put an entire computer on your head. Now it’s a, it’s a little heavy still, but still,

Jeremy Nelson (00:21:07):
I mean, they better, when you have a counterbalance in the back, I said, I find that’s more comfortable for a longer term use, right after 20 minutes, I start to get neck fatigue. Um,

Michael Nebeling (00:21:20):
How long term is actually what we doing long term, like who’s really working in virtual reality. I think I always think of my PhD and master’s students as those it’s been probably the most time, except for maybe some hardcore gamers in AR/VR technologies. And, um, yeah, it’s not comfortable.

Jeremy Nelson (00:21:38):
Well, cool. So we have, uh, you know, the MOOC is about to launch we’re, we’re targeting, uh, in early November to launch the MOOC on the Coursera platform. What’s the process been like, how did you go? I mean, it was changing very rapidly through this whole production. We threw COVID in there, you had to start filming remotely from your house, like of all the possible scenarios. Uh, I think it probably encapsulated just what it’s like to work in this space of things changing and you having to adapt and be agile, resilient. Can you talk about that? And

Michael Nebeling (00:22:15):
So I think my honest, that it’s a little bit like a PhD thesis. It’s like one of those things, although I think PhD was my PhD thesis was probably, that was my primary thing. I had this thing, the XR MOOC was one of many things I was working on in parallel. I was trying to grow a lab. I was trying to manage students’ health, mental health as a lot of things going on. I think we had one or two meetings where my production lead, Annie was probably a little concerned about me because I looked a little sad. So there were some tough times I’m just like, I’m completely honest. Um, and, but I feel like I’m overall very proud when I look at the materials that we have produced and I’m not looking at them. Yeah. Okay. But you had to do it. You had to do it by himself.

Michael Nebeling (00:23:05):
So that’s why it’s like that. No, I’m actually still like, it’s so cool, it’s a HoloLens 2. And I was like working with three GoPros and like two GoPros. I was exaggerating. So two GoPros and half, a few camera angles and mixing into 360 footage. So we’re using the technologies to communicate the technologies. It’s like one of the, I think when I look at a lot of the space in terms of like how these MOOCs are being done there, we’re not the first to do. Uh, uh, we were probably the first to really cover augmented reality in depth. There’s a lot of there’s enough. There’s a few MOOCs out there in the space, some quite established ones that are focused on virtual reality. And some of them come from the vendors directly and some of them from really respectable, um, teachers and universities, so really good stuff. So how do, what, how do we differ with our MOOC?

Michael Nebeling (00:23:53):
Um, well, we don’t do a lot of post production. We actually do live demos, live production. I show we use the technologies to support the teaching and some of the, um, lectures you see me directly in a VR headset and I’m teaching about VR inside of VR headset. That was really fun and really exciting. And, and it just, that was not that our production team went crazy, uh, editing all this stuff in post. Um, no, we it’s this virtual production in live, live production. And I’m very proud of that. I think I I’m very proud of the staff. That’d be, that’d be accomplished. I also liked that I was given so much freedom. There’s a lot of video sequences. I’ve been a hobby video editing person for like many years. I used to do one every around the holidays to show my mom what my year has been like. And video blog has been to use the worst possible video editor and then trying to do something cool with it, see what you can do with it. And now I’ve learned some of the more advanced video editing tools. I’m not as good as our production team, but I’m sure I’m proud of some of the things we produced.

Jeremy Nelson (00:25:10):
You had to adjust. Cause normally you would have a lot more support from the media design team and the, and the folks at academic innovation, but with having to stay, just, you know, provide remote support or you took on a little bit more of that than they normally would.

Michael Nebeling (00:25:23):
No, sir. It was funny how, when you were talking about that support and how do we set this up here in this room is actually, how do we make it work there? And how’s anybody else got to live in this house while I’m like filming it? It was funny how people are so concerned about getting the lectures, right? Like putting a camera and maybe a teleprompter. And you know, that was not the challenge. The challenge was how do you record in an interesting hands on the learner is with me way. It’s like, I’m like, Hey Jeremy, come on over here, that’s driving this car and virtual reality is, I’ll put you in 360. And I think that’s what I liked so much about how we mix these different things together. And sometimes I’m going to say of my students, like I have to style, like to mix different angles to give people a lot of.

Michael Nebeling (00:26:15):
Like this is what’s going on in context. This is where I am right now. And then I like to switch between these different perspectives. Yeah. Some of the feedback. So some of my students are right now watching some of the videos that we produced for the MOOC in my residential course. And the feedback has been very encouraging and I mixed it so that they can also feel free to complain about it. So I gave them other things to watch as well. So they’re not like just like ah Michael wants us to say good things. And I think the feedback is, is encouraging there. One, one point though, is that maybe I’m switching perspectives a little too often.

Jeremy Nelson (00:26:48):
Oh, too much quick cuts. Well, do you have any show us today or have you put together anything that a little, preview?

Michael Nebeling (00:26:56):
So I show you, uh, thanks for asking. Um, so we already covered a little bit of the, I just like do a two minute rundown how I think about it. So, and I’m not even sure this is full screen. And so it’s three courses. The foundation I want to talk about, it’s really about the knowing the shaping and then a lot of the doing both design and development. So we talked about that. Right. And, um, and then I have like some, for each of the three courses. So I took some representative videos from each of the courses that I really, really enjoyed. And this one on the intro, like I show you one example. So when we teach about XR, I think even the small things like bringing in like, uh, a headset to show the learners, Hey, we don’t, we don’t show third party content. Like a lot of the MOOCs are like, okay, let’s show this Google advertisement video about tilt brush. And it’s so cool. No, I personally think it makes a difference if I put those things on my head and I looked like an idiot here, but I have to cut this out. Sorry.

Jeremy Nelson (00:28:09):
Nobody looks great with a VR headset on.

Michael Nebeling (00:28:11):
Yes. So get used to it. And this was just about, look at this. This is a different way of teaching. Now. I’m like inside, inside virtual reality and I’m teaching it, we’re composing. This is a live composition. Um, I see actually, uh, I see the real world through my headset. We have like, I, I have a reference point and then I’m sketching just in 3D and I’m doing something very basic. You’re just starting with the concept, the reality virtual reality continuum that the learners will learn about, but we have like two versions of those lectures. And I’m interesting. So in this version of one, I just showed you and then this immersive one, and I’m interested, really interested to hear, it’s not really a research project at this stage, but I’m just curious for the feedback from the learners. Um, Hey, this is cool, but Hey, this really supported my learning. Right. I’m curious about that. So, yeah.

Jeremy Nelson (00:29:04):
And that’s one of the challenges about, you know, somebody’s watching somebody else in VR, you, unless you’re on a PC based headset with a monitor next to it, you can’t really see what they’re seeing. Right. They have to communicate with you, or you have to just kind of guess if you’ve done that title before, where were they? I liked the way

Michael Nebeling (00:29:24):
I still remember. You were showing me recently. It was very cute. When you showed me, let’s say every moment, collaboration, AR experience, I didn’t even try it. I just teased you. You’re like, that sounds about right. Thanks, man. I, you to see this and you’re not even seriously, the thing is, um, everything kind of still looks the same when you’ve seen it as the most important thing is to see it once for yourself. Right? I think the primary issue right now that I see with my students in the residential course, and I think this will happen to our online learners in the MOOC course as well. You’ll see Michael with all these technologies and it’s cool, but they’re still very expensive. So you’re going for cardboard. Maybe you’re getting the plastic version. Maybe you’re getting an actual cardboard and you’re starting to see a little but it’s, it’s very different from the other things you see in the movies, Avengers.

Michael Nebeling (00:30:19):
And, uh, you know, um, even in some of the most advanced. And so I think my students in the design oriented, course they actually feel like they’re designing in the blind. They don’t have exactly the access to these technologies. So they’re learning to there’s a lot of things going on. They want to learn about how to design for these technologies. They don’t have access to these technologies. I show them lots of video examples. So like learning by looking at videos, but not really experiencing it for themselves. And that is a very difficult position to be in as a designer.

Jeremy Nelson (00:30:50):
Yeah. And as some, if you’ve never tried it, like how do you know what the interactions are? What does it feel like? That’s one of the things at the XR initiative we’ve been, we’ve pivoted over the summer and now making more devices available. So we’re piloting, we purchased 25 Quests to begin piloting checkout for students and faculty this fall. Um, so we’re going to test from Shapiro, checking out some for a couple of the projects we’re working on a couple of the courses just to, so we can get the process down and the cleaning and the, we bought a cleanbox so we can, you know, kill all the virus and then wipe it down and just make sure it’s safe and then begin to iterate on how do we, how do we get more of these devices to students? So I think by next semester we should have that.

Michael Nebeling (00:31:38):
Yeah. We’ve, we’ve talked to instructors. It’s part of the challenge is to get these devices to the students, but even pre COVID, a lot of the instructors still, we just interviewed recently some of our XR initiative involved, a XR instructor, people that have used XR and instruction at Michigan and, um, what they kind of like uniformly, told us that there are still, it’s still a new technology to most students. So having access to it as one, but you’re not like immediately a native, digital native to that or immersive native. I’m not sure what you call this in the future.

Michael Nebeling (00:32:14):
Uh, and so the learning curve is pretty high. So then how do you guide people? But then you’re not with them in the room. I mean, most people will put on the headset the right way. Okay. And that’s not the issue, but then like, uh, the learning curve is still little a little, a little high. Um, and so especially when the instructor has a clear learning goal in mind, but your spending like 20 minutes, like breaking to the students, first of all, getting them to use it in the way you want it. And not them going like crazy, Hey, I can do all this and being distracted, uh it’s, it’s gotta be distribution is one issue and you are addressing that. But then you’re with this challenge where you then want to coordinate remotely and like this, when researchers talk about controlled experiments in a lab where they can control everything, including how, including your tasks and your focus, as opposed to like remote usability studies where like usually the cat jumps into the scene and types on the keyboard. Yeah. So it’s, it’s, it’s it’s but I don’t want to take anything away from you. I think the effort around the distribution is amazing.

Jeremy Nelson (00:33:29):
No, I mean, that’s, those are real challenges. I mean, we talked with professor Hera Kim-Berman and she has secured, you know, 110 Go Oculus, Go headsets for her dental students. And they were similar challenges. Right. There’s just not a, it’s not a common type of device that still needs to be charged. It has to have updates, run you install apps while, while that framework is similar to a mobile phone, you’re you’re on your phone every day, all day. Right. Like, so it’s just,

Michael Nebeling (00:33:55):
Yeah. And I don’t know what my students think of me, but I, um, it’s not like I walk around with these headsets all the time. Um, they’re like toys, there’s like interesting for a day. Really cool. Fancy. And then the next day you’re hoping for a new toy. And, um, I dunno, I’ve always, um, I was a late adopter. I think I got my first touch device in 2011 when the iPhone was there in 2007. So I was like, and I’m an interaction design researcher, always working with the latest technology. So, um, you could say the same with AR/VR or now. Sure. My HoloLens was in regarded in 16, uh, 17, actually 17 and actually came out in 15. Um, but it’s a big financial commitment. So the point is that I think even though you have 130, Oculus GOs the only good for one thing, and then they’re just lying around.

Michael Nebeling (00:34:51):
Right. Um, unless we figure out the network between classes so that they could be, it could be a shared infrastructure. So that, uh, several teachers who embrace these technologies, several instructors, think of ways of, um, there’s a little bit of these technologies. There are a lot of these technologies on campus, but they’re like in different camps. And I think, and then there’s a stock. Like I’m one I’m guilty. I have like seven HoloLens in my lab. There’s just an, a shelf. Sorry. Um, nobody needs them right now on try to give them to some students, but we need to figure out a way to build on a more common platform. It’s very risky because they are evolving all the time. So yeah, I just, I like the effort that I just ordered these quests one

Jeremy Nelson (00:35:38):
And then the midst of it, the Quest 2 came out now I’m having inventory problems because people are taking them off the shelf. I mean, there’s a, there’s a, yeah, there’s a cleaning protocol process there that needs to be addressed just in general, but obviously not for COVID. There’s a visualization studio in the Duderstadt center that had been high utilization right over the last, well, not last semester, but prior to that, and was beginning to gain traction, you’ve taught a few courses in there, right.

Michael Nebeling (00:36:08):
A great resource for my students and they have it’s, it’s become more and more difficult. It’s like a computing resource that you’re like on a, on a schedule. Right. You can, uh, and it’s primarily a student resource, so it’s not designed for instructors to go in there and teach courses in there. And I like that set up. I really respect that. I think it’s good. But when you think about it, you, I walked into this space a few times, pre COVID, but it’s not like everybody is doing AR/VR in, there is games, office documents. You really don’t need an i-7, 32 gigs of Ram and

Jeremy Nelson (00:36:44):
37 inch screen. That’s really nice. That’s fine. Uh, yeah. I mean, I think there’s, there’s a number of the platforms. I mean, how do you think about AR versus VR in terms of the trajectory of where they’ll go from an educational standpoint

Michael Nebeling (00:37:02):
Education? Okay. That’s, that’s a good, uh, that’s a good thing. So, um, uh, I’ll throw in some ideas here. Um, one thing that I’m exploring with my current PhD student Shwetha Rajaram, is the role of paper and how paper can be combined with AR and how, like we often just paper as a marker for marker-based AR and they use tools like before you shout out to Jake and, um, and that’s cool, but nobody really designs around paper. Paper is just the launching the AR experience. They’re not really making use of paper and interesting, useful way. So we were exploring that. I’m really excited about that. So I think AR on the classroom is very cool because I can see my peers. I’m not in this headset. I can see everything around me. It’s also good for the instructor. Cause like you don’t want to be in there. Why don’t you just do this to me? Yeah.

Jeremy Nelson (00:37:53):
Unless you meet them in a social VR setting. Right.

Michael Nebeling (00:37:56):
Maybe a different story. Yeah. So I’m very excited about AR in the classroom and a collocated classroom. I’m very excited about VR in a remote setting. And um, I’m also excited about collaborative AR experiences, um, remote. And I’m also interested in actually, I’ve always been interested in creating experiences where

Michael Nebeling (00:38:16):
in 3D, I don’t care. You can collaborate and you can participate. And I think that is actually the trend. So the trend is not like VR will win or AR will win, the trend is we need to build ecosystems that build around these AR/VR technologies. We need to get away from these approaches where we want to create these islands. These islands of everything is like in my headset. And then the other thing is your phone. But then like they don’t really work together. And, um, I believe strongly that there will always be tasks in the future where VR is the right is the right approach. And there will always be tasks in the future where AR is the right approach. Um, and I think the industry, the industry thinks about, uh, VR is relatively mature, can still do more. AR is still pretty expensive, still experimental. Yeah. Very interesting. I would side with those that see more potential overall with AR I think overall it might be more applicable in everyday life, but I think VR is super powerful for training and simulations. Yeah. So I can’t give you a super clear on this, not one now.

Jeremy Nelson (00:39:33):
I mean, that’s great. I mean, that’s, that’s one of the things we’re exploring with the initiative as well as where, where best can these technologies be deployed and how they can enhance the student learning experience and improve the process. Right.

Michael Nebeling (00:39:48):
And that’s the, that’s the, the number one learning goal I have for my MOC and for us in the initiative is, um, to get a good sense of where are these technologies, good, what are the lessons we can learn from this project? I liked the year, one initiative, a lot like your funding, uh, six to eight projects, depends on how you, how you think about it. Research projects, six of them are these carried out by your team. As far as I understand, the effort is really great. You have hired nice designers and developers. I was a little bit part of the recruitment. Thank you very much. Talal was helping us as well. I think you’re putting it Shalaunda. I think great people you’re adding to your team. It’s really been a joy watching how you’ve been growing this. Uh, I’m in no way your peer, but from an outsider perspective, I think there’s a lot of things I can learn from you, how you, how you, how you grow your team, how you build something inclusive, how you’re bridging. So the AR/VR space is a very interesting space to work in because everybody will tell you that they have been longer in the space than you have.

Jeremy Nelson (00:40:53):
True.

Michael Nebeling (00:40:55):
Okay. I’ve been doing AR/VR when you were a baby man. So, um, and that’s not helpful because we talked about this thing earlier and I didn’t finish my thought. Sometimes it was very useful to have people that haven’t been exposed too much to this technology because they can’t, they don’t think within the constraints they would not just dismiss this idea because no, no, I couldn’t do this with HoloLens, but what if you could, right. Because I didn’t know that you couldn’t do it. I thought it would be a good idea. And that is so powerful. Um, which, um, I think, I mean, anyway, so the initiative is, is a great, the research projects that you’re setting up are all designed right now to bring AR/VR into the classroom and learn potentially my perspective, you should share yours.

Jeremy Nelson (00:41:42):
No, that’s the charge.

Michael Nebeling (00:41:46):
How do we, Can be improved? Learning can even ask the question because the big problem is actually just trying to transform it, right? You do a good job and then can we transform learning and how do we transform it in a positive or a negative way? And a lot of it is trade offs. There’s no people that always present studies to me that VR as much by, better than blah or the other way around. We can’t generalize compared to what you have. You will have six examples. That you can learn from eight that’s eight? Yup. And I think the next year initiative would be interesting. What I would like to hear. I’m not sure whether you’ve given it a thought because he was just wrapping up one. But do you have, am I allowed to ask you questions? That’s it’s two way street. Okay. How do you think about next years year’s goals for the initiative?

Jeremy Nelson (00:42:41):
Yeah, I mean, I think this, this first year was really around building community and understanding capacity and starting to build out some example projects and engage, you know, a wider team and build out the team. I think this next year is about expanding that capacity for content creation. So we, we did a call for proposals last year. We funded it. These eight projects, you mentioned, we are going to do another call I’m hoping to launch next week, uh, early October, uh, but more focused around using commercially available platforms to generate content. So while they’re the experiences we’ve been doing now require, you know, a more complex set of 3D modeling and game engines like unity and unreal and, and deep process. I want to explore some of these authoring platforms for three 60 video or WebXR, our WebGL, and see if we can begin to bring more people into this space to create that may not have to go deep down the development stack, right.

Jeremy Nelson (00:43:52):
And have deep C plus plus or C sharp programming and kind of these longer arc projects. Cause they’re, they’re time intensive. They’re you have to build everything out. Where can somebody walk, where can’t they walk? What can they interact with? Can they go through this wall? Can they go down into the nuclear reactor? What happens when they’re down there? What do we present to them? And I think our team has been doing a great job. Eric and Moeezo have really helped put process and structure in place beginning to iterate, bringing Shalaunda in to help us think about ways to, uh, assess or create the learning goals and objectives, working the faculty to continue to ask them. But all right, now that the student knows this, what can they do with that? What does this help with and why VR and what are we going to do in this space? And it’s really helped us in a pivot on some of our projects to hopefully create a much more engaging, uh, experience for the students and the learners.

Michael Nebeling (00:44:49):
Yeah. And the engaging in this space is still a relatively low hanging fruit. I think you can, you can accomplish that most of the time, from my perspective, it’s also super interesting to, to see some of my students transition into your team over the summer like Toshi, and now she comes back and she has worked with you on some of the projects. And now she is a graduate student assistant in my course. And she’s not only a very lovely person to work with, but she has also grown quite a bit just by working on a range of projects with you. I think this approach of going broad. It’s very good. I, the six examples in different domains, the whole strategy of the initiative to bridge all these different schools. I still think it’s really, I don’t know. I want to say this unique thing, cause it’s like similar to this, like who has been longer in this space.

Michael Nebeling (00:45:37):
We are relatively new in this space, but I think Michigan is doing a really cool job on the education side. Um, we have also been in touch with people who have been studying education in AR like I’m thinking of Iulian at Harvard. I think we’re making the right connections. Shalaunda brought in a little bit of high network strategy wise. It’s just very fascinating how year one was like, I don’t know if you play a strategy game. So year one is where you build your little village. And so your tour is you’re going to attack civilization, no StarCraft, but there’s a bias there. Risk. And I mean attack in a good way because you’re like challenging some of the others in the space. Maybe we can be a little bit more, we have more results to share and show we can be more active with our peers. Right. Um, you’ve been doing a good job interacting with the peers. It says it’s also a lot of work. Like, you know, I think of myself as somebody who was really excited about this space, I want to like do everything and then you have all these opportunities and you’re asking me, Hey Michael, do you know this? You know, this, they want to meet these people. Hey, they might have been. Yeah. There’s a lot overwhelming. And it’s been very good to have you.

Jeremy Nelson (00:46:58):
Well, thank you. Thank you. And likewise, working with you to vet some of these ideas I have and help us shape the direction. I mean, there, there are, you know, dozens and dozens of vendors we’ve been talking with, looking at platforms, evaluating, trying to understand where, where they were making progress, you know, from our perspective, like what can we, should we build, what should we partner out? Like trying to understand where those, those tension points are and will give us the best arc of moving and making an impact here, which I thought you are already. Yeah, it’s great. And then you started to mention students, you know, we’re, we’re hiring students now we’re going to expand. Our students have always part of the mission to, uh, establish our structure and then begin to scale out and bring students in.

Michael Nebeling (00:47:47):
Yeah. Because you want to be a good mentor. Like, um, there’s a lot of students excited about this space. I have to say, increasingly no to students, like initially you come here as a new professor and you’re like, I hope anybody’s going to work with me. And then there’s a point where like overwhelmed and, and, and, and I made a lot of mistakes in terms of like how I approach mentoring and how many students I work with. And sometimes I’ve worked with too many cause I can never say no to people. Sure. That’s not good. That’s I think, I think that’s why I’m so fascinated with how you have approached this whole whole come into this ecosystem where a lot of little islands already been doing stuff. I see you as a unifier. Um, and um, at the same time still giving the credit where it’s due not everything is XR initiative. Brandon, I’m not, we’re not claiming everything. Oh, we understand the role of the Duderstadt center sound an amazing team over there has been supporting a lot of the work we’re looking for ways to collaborate, to kind of like help each other. I think this is actually really cool.

Jeremy Nelson (00:48:51):
Yeah. I mean, I think we can shine a spotlight or provide a megaphone to the faculty or staff that have been in this space. Right. And we can bring that, you know, bring that into light and happy to do that. And like, yeah, we, we can’t do everything. We don’t want to do everything, but we want to build a broader community to help do everything. No, to build a broader community. And you know, we are looking to engage more students. So we’ll be hiring more. We’re going to be supporting a, uh, a design challenge and XR and arts design challenge for diversity equity inclusion, um, for next winter semester. So we’ll have folks that want to use arts and potentially XR to imagine an anti-racist world, um, that they could speculatively design what that would look like. They could potentially visualize that in AR VR. And we would support, um, maybe with providing some equipment or some consulting, access to software, things like that.

Michael Nebeling (00:49:51):
It’s a very powerful space to work in. It’s potentially revictimizing traumatizing. So I have a lot of students that come, uh, come work with some of these, uh, really hard to process topics. I think, um, it will put you in a, in a lot of additional challenges, not just getting the AR/VR right, you don’t want to have a naive solution. I’m so frustrated with some of the things I see out there. It’s like every, it’s like a playground everybody wants to play. And some people build really ugly sandcastles and other people build really nice sandcastles and then they want to own the sandbox and the pushing everybody else out. And, um, it’s, it’s a, it’s an interesting space. Do you, um, like I feel like, do you want to go to some of the questions I’ve seen some

Jeremy Nelson (00:50:36):
yeah, yeah you spurred one point here about where I want to see things go, I want to help either participate in or help shape what an ethical framework could look like around developing. You know, there’s a lot of responsibility. There’s a lot that can be done. You know, we talked about it last spring. Like somebody built a VR experience to revisit her deceased daughter hear 7 year old daughter. I don’t know the moral police, but like, are there some guidelines around what, what could be done? And I mean, there’s a potential to, like you said, revictimize or retraumatize or induce trauma that never existed before, because these experiences are so realistic that their brain doesn’t interpret them the same way as watching a video or a screen.

Michael Nebeling (00:51:24):
I’ve been coming into the space with a computer science background and pushing these technologies fascinated. And then I was shaped by the school of information and the thinking, the thinking there, and the high values of my colleagues that I’m trying to adopt as well around diversity, equity and inclusion. We’re trying to do this at the University of Michigan. We’re not always getting it right, but we can hopefully learn from also these negative examples that we sometimes set ourselves. And hopefully we can hold ourselves to higher standards. A couple of lectures were particularly difficult for me to teach in the MOOC and come up around ethics and design guidelines. It’s really a wild West. Um, I think it needs more people to think about this in this space. I do think that in particular, our school of information students are so nicely equipped. We don’t necessarily need more programmers in this space.

Michael Nebeling (00:52:11):
We now know we need more human computer interaction. We need more user experience design, and we need more people that think about the ethics my students pointed out I was watching. So the one thing I did after the MOOC as a little bit of a celebration, I watched the last dance cause I’m a basketball fan. I don’t look like I’m a basketball player cause I’m tiny. But, um, so I watched last dance and other students were like pointing out, Hey Michael, are you’ve been watching the wrong thing? The social dilemma. And, um, and it’s interesting cause they seemed so shocked about it. So I’ve been watching half of it and I’m like, nothing of what I see so far is surprising to me. Maybe it’s the professorial things, but, uh, you know, um, I knew that that’s like, how can it be shocking to me? It’s shocking that my students think is shocking. That is actually the problem. Not that they put together a show where they bring these issues to, to visibility. What is actually shocking to me is that lot of people don’t think about these issues. And so I need to do a better job as an educator,

Jeremy Nelson (00:53:10):
The awareness of what’s actually happening with all of your data and information. Sure. Yeah. And I, and I don’t claim that, you know, we’re going to set the ethical standard or know it all, but like be part of a larger group that’s participating in that. And we, we were

Jeremy Nelson (00:53:26):
With a participated with the group out of Georgia Tech to help set some data privacy recommendations around XR for this XR safety initiative, give feedback on that, around, you know, FERPA data and personally identifiable information and biometrics as it relate to student records and what our responsibility is in that space.

Michael Nebeling (00:53:46):
It’s interesting. So I went to this, I like the efforts that you’re doing, but I keep saying this, so it might be boring, but I still like it. Okay. So I want to, um, you can not like something’s interesting experience I had, I was giving a keynote. I was giving a keynote recently as exciting young professor giving a keynote. It’s cool that Somebody wants to listen to me. So you go there and I give them this pitch about, Hey, AR/VR, we’ve, we’ve sorted out. We sorted out tracking. It’s relatively stable, not it’s relatively good. We should really focus on these issues of social concerns, privacy, equity, identity, security. These are really difficult topics we need to. And so the questions that I got, I was a little grilled in that, in the conference. It was very interesting. I guess some of my peers wanted to show me who is the anyway, so, and they are not peers that obviously more advanced than me like full professors and that.

Michael Nebeling (00:54:41):
So is this a little bit, it was an interesting pre tenure experience, I guess anyway. So I thought I was surprised to get the pushback. It’s like, Michael, are you sure that we have solved tracking and sensing? Shouldn’t we be spending more time on those, but I’m saying, yeah, yeah. I mean, we have the fundamental, but the next level is to do tracking and sensing in a privacy preserving way. I can build the craziest infrastructure and sense the hell out of you, but that is not what I should be doing. And there comes, this other thing might be, I don’t know you triggered. So this project that you mentioned with the daughter deceased, I had a student walk into my office. I lost my brother in 2008 was a difficult time. So what I want to bring him back in, in VR in 360, how would my mom think about that?

Michael Nebeling (00:55:31):
So I was thinking a lot about this and I decided with a student, I don’t think it’s the right project for me to work on emotionally. And that student was going, I had a similar past. So we thought we connected, well, we decided not to do the project. So now somebody asked us the project. So now I feel like not like, Oh, somebody stole that idea. I feel like maybe I should work on it. Maybe to set the right example because a lot of things went wrong there. Um, and so I feel like I actually missed an opportunity to, uh, um, to sacrifice a little bit of my own mental health, maybe for generating something that maybe might be useful. I might also be wrong with how I think about, but it’s like one of those topics that’s really close to you, you feel that I know how to design this. Right. I feel like I can guide the student that I should have done this. And I was sad. I made it as one of the examples

Michael Nebeling (00:56:24):
In my MOOC where I was like, yeah, I made the decision not to work on the project and this is good for those reasons. And this is different for those.

Jeremy Nelson (00:56:32):
Yeah. Well, it’s good that you have the perspective to look back and ask, what could you have done? What might you do different going forward and perhaps change? Well, this has been an awesome conversation. You know, we could talk for hours. We do sometimes. Uh, I appreciate you coming back. I’m really looking forward to the MOOC launching in November. We’ll be sending out marketing about that to everyone. Do you want to take some questions now? We have a couple of,

Michael Nebeling (00:57:01):
Yes, please. I think we have a lot of patient. Yes. Visitors. Uh,

Jeremy Nelson (00:57:06):
I think we can open up the lines if folks want to. Was there Trevor, was there somebody that wanted to chat? Do you mind helping us with that?

Michael Nebeling (00:57:16):
I’m reading three posted questions. I’m reading through them right now.

Jeremy Nelson (00:57:24):
Is it Jatin or how do you, sorry. Okay. Hi.

Guest 1 (00:57:28):
Yep. That’s Jeff. Jatin hi. How are you? Good. How are you? I’m great. And I hope I am already like, yes. First of all, amazing session. And I had some of the best insights I’ve ever had in. They must’ve been zoned as of now, did you pay him? I did not pay anybody. Nobody’s paid, you know, this is why we do this thing. So I don’t know how many other people are here because I cannot see that. Okay. I am from India a hundred thousand .

Michael Nebeling (00:57:57):
Okay. It’s so cool that you’re joining us remotely from India.

Guest 1 (00:58:01):
So I am from India and I just graduated from my college. So that is university. And I just entered my new job. And I’ve been walking into this domain for the last two to three years from my second year or so, or my name I saw. Okay.

Guest 1 (00:58:13):
So I had a, I had, I have a lot of experience with not as compared to you, but the main questions that I have is first of all, it’s amazing thing that you would just mention, you know, that the use of people has been just limited to just as a market. Okay. How can we, you know, make it more fruitful? So what do you think are some of the most vaguely.

Michael Nebeling (00:58:33):
possible ways we can do? So, so one of the, so I wish Shwetha was on here with me, she’s the student working with me on this. One of the things we are really excited about is the idea of creating interactive handouts so that we can give a paper based hand out to students and we build the tools. So that instructors are in a position to create those handouts actually using paper-based techniques, not like a 3D Authoring tool that they have to learn.

Michael Nebeling (00:59:00):
First. This is barrier to entry. And, and so I like we’ve explored like six different examples to has a lot of interesting around and like collaborating around paper. It’s very interesting. It’s very versatile. You need to think about what’s the AR perspective. Is it, is it going to be a headset? Is it going to be a tablet? Is it going to be a smartphone? So we’ve explored all these. We’ve actually played with the HoloLens 2 the data tracking there. Can we track ends? Can we track hands? How well does this work? When you hold a pen, you’ve looked at iPad with LIDAR, basically built in 3D scanner now, um, that doesn’t have a lot of the, um, uh, sense-making yet, but I mean, it has a very good read of the geometry of the environment around you couldn’t distinguish paper from the environment.

Michael Nebeling (00:59:43):
So the precision isn’t. And so I think it’s an interesting space to play around with. I just don’t want to just share this one interactive handout example. It was, uh, around, um, physics and explaining how forces work and should that came up with a lot of the examples. And I think it’s a place where a lot of people can still play with, because there’s this book on the myth of the paperless office, which is true. We use paper a lot. I’m not sure whether you would agree. You use a lot of paper in your education right now. We do see. So then I don’t think it’s all going to go away with AR/VR. So we need to think about these integrated ecosystems. That includes analog devices, like paper, probably it doesn’t think of paper as an analog device, but it is.

Guest 1 (01:00:33):
Yeah. Yeah. Okay.So the reason that this question arises to me, because I’m also also working into the domain of education for using education, I’m working in a startup. I just joined, I am sorry. The name of the startup. So here I am a research design. Okay. So my sole aim is to build experiences, which are out of the box as given to me is to, you know, uh, relate this R of your experiences with the reality. OK. Whether it may be AR/VR, we are still, you know, interacting with the virtual objects. Okay. But now my task is to, you know, make these virtual objects interact with reality. Okay. So what do you think are the things possible? Like this one, you know, the paper experience is a good example. I also thought on it and I was, you know, the simplest experience that I could think of was that there are two, you know, what’s your virtual object.

Guest 1 (01:01:20):
One is a battery. One is a bulb and the user has to join the connection, why up insert, it draws on the pencil, on the paper, using a pencil. And once the connection is made, the light bulb turns on, okay, this was one of the things, another thing that I, you know, tried to build was a blow air experience that was for, you know, giving the user the understanding of the forces. So basically in that experience, you will spawn a boat in the ocean which is virtual, and you would have to blow on the screen of your smartphone. And depending on the force of your blue, the boat will move ahead with that force.

Michael Nebeling (01:01:54):
That’s cool. And you’re just doing this on the microphone microphone.

Guest 1 (01:01:58):
Exactly. Exactly. I took the microphone, readings, you know, there is, some, you know, clamping and stuff like that. And we mapped it and then okay, local force in Unity. But that’s done. So what other things are there? Other things. Okay.

Michael Nebeling (01:02:13):
So first of all, I think I’m talking to somebody who is really creative and explorative, uh, and probably

Michael Nebeling (01:02:20):
Still, uh, I don’t, I don’t, I can’t see you. So a young maybe student, but has all those really powerful energy. Um, you sound like really amazing person to work with. So that’s cool. Um, I do think that like, what are you actually asking? Is it, wouldn’t be helpful if I, if I brainstorm six examples with you now, which we could probably come up with, but you should think about your method. What is the method? We explore that design space of interactions. And I think you often look at example papers. I’m not sure I’ll be talking about academic research. Are we talking, are we talking about it from a brainstorming, uh, perspective from an interaction design perspective? I think you want to, you want to set yourself as a few constraints for how you think through the design space and you were bringing in the microphone, the phone, um, I have all these sensing things.

Michael Nebeling (01:03:17):
So you’ve just designed a very big space. And then paper is also in there, like, um, is an endless design space, uh, and maybe techniques. And so I think it’s a methods. What you’re actually asking is like, what’s the right method. Not like, not what, a five other examples. I think if I give you the right method, which I, I can’t come up with that on the spot right now, but I think method really approaching this systematically. So I have a lecture in my MOOC on brainstorming and ideation and problem framing. And I think right now, what I hear is like so much creative energy, but it needs to be channeled in a interesting way. And often I don’t know what you are most interested in is I think is the right way to channel it. But then think about as few different variables you want to play with maybe maximum of three, if that helps you in thinking about this abstractly and then thinking through all the different combinations, that’s what I would advise you to do. And you may not be happy with that answer, but it sounds like smart. No [laughs].

Guest 1 (01:04:29):
I mean, I don’t know if I’m asking many questions. One last question that I would like to ask is where can I, you know, read? What about the things that you guys are doing? Are you, you know, presenting them are the researchers about it? Are there any videos about it? You know, I would be happy to go through them if that’s possible.

Michael Nebeling (01:04:45):
Yeah. So, I mean, you can find some of us online, Jeremy, what do you think?

Jeremy Nelson (01:04:50):
Yeah. I mean your, your work, I mean, you could give out your lab website, we’ve got a website for the XR initiative, where we point to some of these other locations, we started a podcast, obviously this is one of the live recordings, but, uh, we’ve recorded, you know, 25 or so episodes. We’ve, we’re launching one a week. So that’s a place to learn more about what’s happening here.

Michael Nebeling (01:05:11):
It’s cool that our first, uh,

Michael Nebeling (01:05:13):
Person is remote and feels so close at the same

Michael Nebeling (01:05:18):
Time. So I wanted to thank you for your, for your questions and keep up this interest. I think this is going to be very in the way.

Guest 1 (01:05:28):
Thank you. You already go, I do run a community myself by the way, because three years ago I started out stuff. I was in, I was finding people, but I was not able to connect with them. So I ended up starting my own community with four people only. And now we are like 7,000 plus people strong. And so that is what we are doing. And, you know, I would love to be, you know, how can we as a community, maybe can play a part in it or whatever, you know? Sure. I’m really excited about the stuff you guys are doing. And I’m really amazed.

Jeremy Nelson (01:05:58):
Well, we’d, we’d love for you to share the, the MOOC when it comes out. I think that audience could be very interested in that.

Guest 1 (01:06:06):
Thank you so much. No more questions.

Jeremy Nelson (01:06:09):
No, no, no. It’s great. We’ve got we’ve I appreciate that. Uh, yeah.

Michael Nebeling (01:06:13):
Maybe we can even follow up at some stage. Yeah,

Jeremy Nelson (01:06:15):
Yeah, yeah. We can follow up. I see, there was a question from Jake. I think he had to jump off. Probably has there been greater interest in XR across the university, particularly from the faculty now that remote learning is the norm, at least for awhile,

Michael Nebeling (01:06:33):
I find it very interesting that people ask those questions. Like, is there a way to say no to that one?

Jeremy Nelson (01:06:38):
I mean, the interest is there, right? Like I think it’s just a little,

Michael Nebeling (01:06:42):
Is Jake, is it a revenue driven, Jake? You’re doing the right. You’re working in the right space. Yes. A lot of interest. Um, the challenges are still very real. It’s, didn’t get easier. It actually made a lot of things harder. So what do you think?

Jeremy Nelson (01:06:56):
I agree. I mean, the equity and the devices, the platforms, I mean, there’s a lot of innovation going on with these collaborative, you know, VR or, you know, things like spatial where you can connect in AR or VR and have a more 3D, you know, zoom meeting. Um, I think there’s, I think that will evolve, right? That will happen. There will be more presence based collaborative meetings like this, you know, where it’s, I think there’s a lot of zoom, but a lot of, you know, it’s still 2D and you still got these phone on

Michael Nebeling (01:07:31):
My, so I think it’s relatively easy to do something XR in a, in a, in a, in a class there’s a lot of us, uh, experimenting with just bringing in something, XR into our classrooms. There’s a big difference in, in designing meaningful AR/VR experiences, tied to learning goals. And that’s why I think it needs, it needs that infrastructure that you have. It’s so smart to put you inside academic innovation to be, to have an access to a faculty that worked directly in educational research. And, and, um, and, and that, uh, we experience these challenges every day. Like, um, like I, one thing I’m thinking about is how could we, so we

Michael Nebeling (01:08:16):
We’re doing that at a conference next year. And every time somebody organizes a conference, they always want to kind of like add one new thing, push, push us to the next thing and Steve jobs. Yeah. So previously it’s been a, yeah, you couldn’t really innovate because obviously you had to deal with COVID-19. It was like last minute change we understand. But I think the expectations for next year’s conferences are like

Michael Nebeling (01:08:41):
Really high. Like those that, that

Michael Nebeling (01:08:45):
We just have to assume it’s going to be mostly virtual. So, um, how are we going to innovate? How are we going to make it an interesting experience for, for the people to attend and beyond zoom I’m hoping?

Jeremy Nelson (01:08:59):
Yeah. Right. Platform. Why I was talking with our faculty in school music today, that’s attending a conference and he said, they’re doing the receptions in Altspace. He said, could I borrow a VR headset so I can participate in those? Um, so they’re, they’re trying out the networking receptions in, you know, a virtual collaborative platform. Uh, I’ll be curious to see how it goes. Right. But there was a, we did a quick training session on how the device and how to use it and, you know, I’ll be available to help, but, you know, that’s, that doesn’t scale. Right. I can’t provide one on one support for thousands of people.

Michael Nebeling (01:09:38):
And, uh, uh, the big, the theme of our conversation has been around scale. How do we measure success? I think that’s what I hear in a little bit. When I listen to ourselves, uh, Ruchita, who has been a student actually from, I still remember you, um, ask us, and I’m going to, I’m going to take the question, but you’re going to take the onset, uh, the most pressuring, ethical issues in AI today. What do you think there’s, there’s a whole,

Jeremy Nelson (01:10:08):
There’s a whole discussion on data privacy, what can be done with this data. I mean, I think the, the number I’ve heard is in a 20 minute VR experience to generate 2 million data points on average, but what can I collect? What ads and what the transparency into the information who’s housing. It, you know, the, the company that owns Oculus is Facebook. You know, they have not always been transparent with what they’re doing with the data. And so I think having, uh, you know, providing easier ways for users or people to see what’s happening with your data, being able to opt in or opt out of sharing. I think creating, I think the GDPR went a long way to begin to make a change here. Cause there haven’t been policies at the federal level, in the United States to do anything about that. Um, I think that for sure, around the data, privacy and

Jeremy Nelson (01:11:14):
Security, and then as we get into that, you know, trauma or what types of experiences you can create, I mean, you’re, I don’t know if you want to talk about that. Have you done much work? I mean, Jeremy Bailenson, Stanford’s done a lot of work around how your brain interprets these VR experiences and you know, his message like don’t do, don’t do bad things in VR that you wouldn’t do in real life because your brain doesn’t always know the difference.

Michael Nebeling (01:11:41):
Okay. Um, I like, uh, any effort that tries to provide frameworks for, to think through this space, like just, uh, one of the issues you just saw with an earlier question was like, the space is so complex. How do you think for it? How do you know that you’ve sufficiently thought through it? Um, and I think understanding that designers and developers have a responsibility. You’re not just coding, right? Your responsibility is not to create, get the most data out of the users to fingerprint anything and everything that you can then sell. That’s not your responsibility. Um, you have a, you should be happy with what you’re creating in this space. Sometimes it can be experimental. I I’m not against when it’s clearly defined as such any kind of deception and user studies. I think there’s, there are good reasons for it sometimes, but I actually think one of the best ways to approach this whole space as to, I mean, the universities have been in establishing these IRB, these institutional review boards that are looking at research and anything that involves human subjects and potentially creates data and AR/VR is all that. I feel like everything we do in AI, VR is human subjects research and, um, and there need to be these, uh, these processes in place. Um, it’s not good enough. This idea don’t do to others. What does they call don’t do to others? Would

Jeremy Nelson (01:13:12):
You don’t want done to you? That’s that’s very subjective, right?

Michael Nebeling (01:13:16):
Yeah. I think some of us don’t care, so it’s not the right ethical framework. Right. Otherwise you wouldn’t have so much crime in the world, so, um, yeah. And we need more. One of the big issues is if we don’t have a lot of way, a lot of good, uh, well articulated frameworks to think through this. And I think what you’re calling a standard might be, uh, might be one of those examples. Um, the problem is adoption later.

Jeremy Nelson (01:13:44):
I mean, what about, I mean, we haven’t even, would you touch on it briefly? And it re crystallized to me in the Facebook connect presentation a couple weeks ago is the work they’re doing around their AR glasses and sensing the world, like what you’re wearing a device on your head, that’s sensing everything, potentially recording the world. And they were landing where you, like, you’ve got to turn it off when it goes into the bathroom, how do you know you’re in a bathroom? Like what, like you want it scanning and sensing and recording. There are just areas to address.

Michael Nebeling (01:14:17):
Whereas when I work with these technologies, there’s two Michaels working with these technologies. One is Michael, the researcher, and one is Michael, the, uh, private, uh, being. And, um, as a researcher, I am okay to take more risk, but I, uh, approached us with a, like this IRB perspective. Like I want to, I don’t want to get that data at every cost. Like I want to make it that the risk should really be in line with the benefit to the user. Their risk should be minimal. I think that that, and risk is like really seeing these risks is I think is like, just don’t understand. Don’t appreciate that there are a lot of risks to it. Again, this project that I work on with Swetha, that was really interesting to ask about whether people would use a hollow HoloLens in their home, participate in their educational scenario, but all they do is work on paper, but they were all about, and we generated meshes from their environments and we showed the users, those meshes.

Michael Nebeling (01:15:14):
And it was very interesting. There’s like, okay, you can’t see anything. Their resolution is so bad of these 3D reconstruct, I don’t care. And so then you asked them, but what if I showed the whole video feed to people and then like, Oh no, that that’s, maybe I w I wouldn’t want that. And so then I’m thinking, okay, the problem right here is that you do not understand that what I told you actually extra data, that I can take you home, the key to your door. And, and that is alarming. But I think, um, not a lot of people make those connections yet. They’re thinking, um, there’s the cynical present thinking about AI. VR is not mature enough. It’s not going to go anywhere and blah, blah, blah. Um, and that’s when you see those examples or it’s potentially very powerful or dangerous,

Jeremy Nelson (01:16:14):
So, well, then you begin to add AI and machine learning on top of that. And now I begin to predict and make assumptions about, especially in learning, right? Oh, you’re already, you’re not getting this challenge in VR yet. You’re going to fail. Right. Everyone else that has done. It has failed, you know? And so then that gets into the whole algorithms and, and that another conversation, uh, we can have another episode about that. There’s a, there’s another message here from Khalif. I think I’m a mechanical engineering student, junior at the university, and I’ve been looking into building my own AR platform. How do you feel the current AR/VR platforms hinder people from accessing them

Michael Nebeling (01:16:54):
From accessing, um, accessing

Jeremy Nelson (01:16:57):
The platforms or creating content? I would assume, unless Khalif, do you want to, are you still here? Do you want to,

Michael Nebeling (01:17:04):
I think we’re running low on participants. That’s okay. If you still have more than 10, that’s good. Uh, so it’s not just you and me, but we can keep going. Uh, yeah. There are a lot of challenges actually there. So maybe, I mean, so it’s hard to structure the answer that you’re looking into building my own AR platform. I think that isn’t yeah. I wonder. Uh, so that is an interesting direction, right? If you’re not happy with what the people do, should you be building your own platform? I’m, I’m thinking that’s a scenario where I’m thinking, you know, we need to give some credit to some of the things they are doing because, um, they, on the technology side that it’s not like Facebook doesn’t understand what the technology can do. They’re just really taking that risk. Sometimes there’s also this constant probing and testing.

Michael Nebeling (01:17:53):
How far would people go with sharing their personal lives on Facebook? I mean, people process all kinds of things on Facebook, success and failure, life and death. And it’s been very interesting, this whole transformation of the social media space. And like that’s just side with them for a second. It is potentially fascinating to learn how far people would go, because we can potentially offer you these advanced services. We need to tell you, it’s like the trade offs you need to make, because you need to be able to understand when you get these advanced services, I have access to this data. And then there needs to be a contract where you agree what happens with that data. And I think, and I’m not even sure whether I get this right from a lawyer perspective, but I feel like that should I understand that they’re trying to push things. And I also think that’s the way to do innovation, but not at all costs. And this informed consent is a very, very important. Yeah, sure. I answered the question directly, but I feel like maybe you want to cover some of the others as well. Uh, I don’t see the questions want to do.

Jeremy Nelson (01:19:03):
Yeah. Yep. I think we’ve answered all of on there’s a, there’s a silly one from Eric is XR the silver bullet to all the world’s problems. No.

Michael Nebeling (01:19:13):
Oh yeah. I agree with you the next one. Thanks, Eric. Yeah.

Jeremy Nelson (01:19:18):
Keeping it light. Yeah, I think, I mean, the question about build or buy is important. Like what, what I would say in that, what are you, what problems are you trying to solve? What issues are you trying to address with building a new platform and where do you want to take it? You’re trying to solve

Michael Nebeling (01:19:37):
It is interesting. Do you remember the, Oscar, like this idea of creating a social network that is a distributed, physically, all the nodes are distributed. Nobody owns the actual data and isn’t, isn’t a Bitcoin in some ways set up the same way. I don’t actually know exactly. I don’t understand all this, but blockchain, blockchain, blockchain. Um, but it doesn’t gain the traction that you would think it’s like people still coming to Facebook and maybe, I mean, it becomes also like, why do people vote the way they wrote? We need to, we need to, um, a lot of the issues are, I think a lot of the issues start with us in education that, um, AR/VR, shouldn’t be this thing that’s only done in a few places. No, it’s there. It’s right here in to build it into every curriculum. Kindergarten kid needs to know what there smart phone can do conscious by one when you’re seven.

Michael Nebeling (01:20:34):
And then when you’re 18, you learn about what you actually did for the last. So, you know, this education needs to happen very early. So that means that our instructors need to feel more comfortable with these technologies. A lot of looking at that very exotic as like, Oh, we need that. That’s the thing we that’s the role that we can play. And that would add to the ubiquitous question from Christian here, how big it is. Do you think XR technology will be to the average president in 10 years, he’s referring to a specific hyperreality video that is still dystopian bad vision of the future.

Jeremy Nelson (01:21:13):
That video was quite dystopian. When you got hacked, a lot of people wanted to put something on their face, right? There’s all these rumors of Apple building these glasses that can create a there’s still problems to deal with. Right. Heat and computer and optic, like there’s there’s physics problems, uh, that need to be dealt with

Michael Nebeling (01:21:44):
[inaudible]

Jeremy Nelson (01:21:45):
Contact lenses that put a computer on your eyeball, right? Like that’s even closer. Yeah.

Michael Nebeling (01:21:48):
So when people say XR, like the question is, what do they actually mean? Because like, uh, you know, there’s this argument to be made around smartphone based AR and how will you be good as it is already? Like, I mean, I’ve been talking about several hundred millions of users. They could have it right now. Do they use it? I don’t know. Do they see the value in it? Well, maybe we don’t have the right content yet. And, um, how much more you will be in 10 years? Well, in terms of numbers, like maybe we have more people with headsets and they over a few more people, many people have a smartphone just in the world, but we don’t have that. And I haven’t checked the latest numbers, but I still feel like it’s pretty, it’s relatively pervasive. A lot of people have some of these latest generation smartphones. And you could argue that. I mean, the penetration, I’m not sure what the right words are. And this marketing world is, is, uh, there are people who have access to AR um, already getting access to VR is difficult, more difficult.

Jeremy Nelson (01:22:52):
Yeah. There was a question about AR and VR. I have a place with home assistance IOT. I think so. I mean, there’s a group I talked to, they talk about ambient computing and they take it a step further of how AR and VR

Jeremy Nelson (01:23:06):
Plus IOT devices and all these sensors kind of around you are all working together to create a digitally enhanced space. Right. I mean, Elon, Elon’s building a brain link interface, right. To plug you into the matrix. I mean, they’re, they’re starting, I mean, they’re starting with people that have physical disabilities. Right. But it’s, it’s not hard to see where you take it from there.

Michael Nebeling (01:23:33):
Yeah. The pitch is good right now, but is it, is it, is it potentially a weapon? Right. Um, so yeah.

Jeremy Nelson (01:23:40):
Last question, maybe what do you think web AR will replace? Application-based AR

Michael Nebeling (01:23:46):
Um, I’m a very strong believer in web XR, but I do not think it will replace. Um, the, so the question is like, um, the native mobile based AR versus running AR for my web browser. Um, if you ask yourself seriously, how many web apps you use on an everyday basis and how many actual apps are you using and how often do you do online banking on your phone, through a website and how often, if you do online banking on your phone, how often do you use an app? There’s this certain threshold that everybody has? It’s like, Oh, no, no, for this, I need to use a real app. Like also think of the seriousness of the task. Like if you’re booking a trip, do you do that on your phone? Or do you go to a powerful computer where you can see the bathrooms of the hotels that you’re like?

Michael Nebeling (01:24:33):
Yeah. So, um, I, my answer is, um, while I’d like to think that the web XR, our role will increase over the next few years through due to distribution. And it’s actually really easy to distribute content this way. I don’t think it will replace the AR or VR app markets that we have, but they need to work more together. These app stores. I hate those sidles, those islands. I understand marketing, I get it. Apple wants to own this and Facebook wants to own it and go, come on, guys, I’ll be building a better world together, or are we, what are we doing here? And I think we could be, it could be building a better world together.

Jeremy Nelson (01:25:20):
Thank you so much for spending time today, giving your thoughts, Michael, and talking about your MOOC and helping shape the XR initiative. Thank you all to all of our guests and attendees and questions. It was a great discussion and I really appreciate it. Uh, we’ll be having another event, uh, October 29th and a virtual platform called Virbela. So look for messages coming out about that for our XR speaker series this fall. I should be interesting with Michelle Aebersold and nursing talking about her, Getting Under the Skin VR simulation, and we’ll be doing it in a, a bit more immersive, even though it’s on a application on your computer. I mean, I couldn’t be more immersive. Yes, yes. You can’t get past this wall here. My screen. Uh,

Jeremy Nelson (01:26:04):
Well, thank you so much. It was great talking with you. Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at ai.umich.edu/xr