Jeremy Nelson, Director of the XR Initiative

In this week’s MiXR Studios podcast, we talk with Dr. Prashant Mahajan, the professor and vice-chair in the Department of Emergency Medicine; Professor of Pediatric Emergency Medicine; and Division Chief of Pediatric Emergency Medicine at the University of Michigan’s Medical School. Dr. Mahajan has been an early adopter of XR technologies within Michigan Medicine to explore how augmented reality could be used to assist emergency medicine practitioners in treating pediatric patients in the emergency department. In this episode, we discuss his work with the HoloLens and some of his exciting work aiding the healthcare team to save lives.

Dr. Mahajan’s journey into XR started with a video of the HoloLens 1 device that led him to other videos and ultimately to the work that Case Western has been doing with their HoloAnatomy course. As a physician in pediatric emergency medicine, he has always been curious about how technology can assist healthcare professionals in their practice of saving children’s lives. He immediately saw opportunities to explore where VR and AR could provide just-in-time training. While VR can provide valuable simulation experiences and training, he wanted to see what could be done with AR and the HoloLens to provide real-time decision support for physicians and if it would improve care or add to cognitive load.

Dr. Mahajan was an early adopter of the Microsoft HoloLens through a generous gift from Kent Foster, director of university relations at Microsoft. Microsoft donated more than 30 HoloLens 1 devices to U-M in an effort to seed innovation and research on how their Mixed Reality technologies could change the world. Dr. Mahajan partnered with David Chesney from the College of Engineering, working with Chesney and his students to explore how this technology could be used in pediatric emergency medicine. The goal of the partnership is to provide students an opportunity to work on life-saving solutions with subject matter experts at Michigan Medicine. This powerful collaboration demonstrates the unique opportunities that students and faculty at U-M have to work together to build something cutting edge. The first year of that collaboration was spent ideating on ways this technology could be used, eventually settling on how to provide remote assistance for rural Emergency Departments with the Hololens from specialists at Michigan Medicine.

Our conversation progressed into the work Dr. Mahajan is doing with the HoloLens 2 to develop an application to provide real-time support to physicians performing pediatric resuscitation. The goal is to determine if this technology can assist clinical decision making at the point of care and understand if it could be used to upskill or reskill emergency medicine physicians that aren’t trained to take care of children. As Dr. Mahajan mentions, 85% of children seen in emergency departments are not staffed by people trained to take care of pediatric emergencies. If this technology can provide that support, lives can be impacted and even saved. Currently, his team is working with a company out of Grand Rapids to create a HoloLens 2 application that will provide real-time assistance to physicians and measure metrics around timeliness of diagnosis and intervention to qualitative measures around stress and cognitive load.

We discuss his thoughts on the current challenges and risks with XR technologies and how evaluations must be deliberate and measured. There are many vendors in this space, and while the marketing videos for the hardware and software are compelling, it is imperative that we take a rigorous approach to comparing these technologies versus other methods. When patient lives are on the line, there is an even greater responsibility to follow research methodologies. Dr. Mahajan’s primary goal is to demonstrate that we are not harming patients with these interventions and technological assistants.

We conclude our conversation with a discussion around the future of the XR Initiative and how we can collaborate more effectively with teams with diverse backgrounds to support the University at large. The broad expertise at the University of Michigan from Architecture to Engineering to Information, Nursing, and Medicine, just to name a few, creates a unique opportunity for students and faculty to build something transformative and sustainable. We discuss the importance of creating more capacity for content creation and how to reduce the time from ideation to application.

I enjoyed my conversation with Dr. Mahajan and appreciate all of the work he is doing to help patients and providers and to push the limits of this technology. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 19

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson. And today we are talking with Dr. Prashant Mahajan, who is the Professor of Emergency Medicine and Pediatrics, and the Vice Chair of Emergency Medicine and the Division Chief of Emergency Medicine at the CS Mott children’s hospital. We are talking about his work with augmented reality and how his team is experimenting with the HoloLens to aid in diagnostic support coming up next in our MiXR Podcast.

Jeremy Nelson (00:42):
Welcome. Thank you for joining us today.

Prashant Mahajan (00:45):
Thank you, Jeremy. It is a pleasure to be here.

Jeremy Nelson (00:48):
Yeah. I’m excited to be talking with you and having you share your stories of, of your XR journey and, and what you’ve been doing at Michigan Medicine. I know it’s such a, an emerging space and so many opportunities for XR in medicine. Love to hear how you got started in XR, kind of what the roots of that were and where you began.

Prashant Mahajan (01:12):
Sure. Um, so I came across this whole concept of, uh, uh, extended reality. Uh, I think that time it was still called augmented reality because I came across a video on YouTube, funnily enough, about the HoloLens intrigued by what it could do. And, uh, and you know, how you, uh, you link up one video to another and I got more and more informed and that’s when I connected with Microsoft. And then I got to learn more about what augmented reality is. I found out the nuances between virtual and augmented reality, and then started learning more about this whole concept of extended reality. And now the terms have even morphed over the last four years to spatial computing and ambient computing, you know, so it’s a very intriguing to me. Yes.

Jeremy Nelson (02:02):
Yeah. So what was that initial, that initial interest? Did you see somebody doing something? Did you see something outside of medicine that triggered a thought?

Prashant Mahajan (02:11):
Yeah. So what happened is when I looked at the initial video about HoloLens, and then they linked at that, uh, video, there was another link to how Case Western, I think is using the HoloLens to teach anatomy. It just struck me that, uh, if you can really augment an individual’s, uh, physical environment, then it could substantially enhance, uh, the way you make decisions. Uh, and just as a quick background, because I work in the setting of emergency care, uh, you know, as a pediatric emergency physician, we know we have so much time constraint and we have to make rapid decisions, and sometimes you can make an erroneous decision. So one way to help that and increase patient safety and obvious health outcomes is if you can assist the clinician with decision making. And what intrigued me is if augmenting the environment to enhance decision making would be something that I was looking for. That’s how I got started into this.

Jeremy Nelson (03:13):
Yeah, that’s fascinating. So what types of challenges were you, were you facing or were you seeing that this technology might be able to help solve?

Prashant Mahajan (03:24):
So, so one of the things that, uh, struck me is, uh, how healthcare is delivered, right? So in the sense you become better and better at doing what you do, if you get exposed to it again and again. So for example, a surgeon who operates on a appendectomy say, hundred times, you know, for instance, is much better, uh, in the outcomes for the patient, uh, as compared to the person who just does it once or twice, or who does it once and then does not do it for five years. And then does it again, you know, this whole concept of training and skills, uh, retention, uh, was, uh, is very important in medicine and more so, especially when you’re constrained for time, you know, like in an emergency situation. So a patient is brought into the emergency room and you don’t know what the patient has and you resuscitate that patient, but then what do you do if you have not resuscitated a patient for a very long time?

Prashant Mahajan (04:24):
So it adds a significant level of stress. And most importantly, the patient outcome is not good if you, if you make a mistake, right, your muscle memory is not kicked in. You’re not trained at that time. So I think there is a possibility to enhance that, uh, by, by using some of this technology, you know, uh, and that’s where I believe is the current limitation of our healthcare provision. And I just give you a simple statistic saying 85% of the children in the United States for instance, are seen in emergency departments that are not staffed by people trained to take care of children, pediatric emergencies. Hmm. Imagine the Impact. Right. So that, that is one of the things that I felt that, uh, probably, uh, technology can assist us if there is a good enough technology. Right.

Jeremy Nelson (05:15):
Sure, sure. So you’re, so you’re imagining that the clinician and the physician would be wearing a HoloLens. So what sort of they’d have access to the diagnoses they’d have access to trainings? How did you imagine it? Or what did you think might be possible?

Prashant Mahajan (05:33):
Right. So, uh, so one thing I would like to just clarify is because, you know, I was looking both at virtual reality and augmented reality. And even though I liked the concept of virtual reality, what bothered me is because you are cut off from the real environment. So the scope of virtual reality was largely, at least at that time was, you know, training people in a, on demand sort of a setting, but not necessarily in the real world context, at least that’s what I initially felt. Sure, sure. Augmented reality. I felt you could really impact, uh, the care. So the way I was starting to look at this was, can we use this for training people? Can we use them for doing something called just in time training? You know, just before, just before the procedure is performed, I sharpen up my skills, I revise it, I review it and then do it. And then finally the Holy grail would be actually doing, providing real care and assistance in decision making in real time.

Jeremy Nelson (06:33):
Sure. Yeah. No, that’s, that’s the future, right?

Prashant Mahajan (06:36):
Correct, Absolutely.

Jeremy Nelson (06:39):
Yeah. So, so that’s exciting. That’s really interesting. So, so you’d reached out to Microsoft. I know they have a, you know, a history with the University of Michigan and a long relationship, and they donated a number of HoloLenses across the university. Did, did you and your team get some of those devices or how did, how did that go from there?

Prashant Mahajan (07:00):
Sure. Yeah. I just want to, um, say, um, that, uh, you know, I’m really very appreciative of Microsoft because they, they were very open to collaboration, so that was great, you know, and they were open to, uh, risk taking ideas. And I don’t think they had looked at use of, um, the HoloLens, uh, in decision making in extremely austere circumstances like the emergency department or on the field. Right. So when I approached them, they were very enthused about it. And then they donated a very generous gift of HoloLenses and, uh, some financial support, you know, uh, to push forward to this idea of decision making in emergency care. So that’s how I got involved with Microsoft. And they’ve been amazing partners since.

Jeremy Nelson (07:47):
That’s great. Yeah, no, there’s just, there’s nothing like having the device and some support to show how the software works to really jumpstart and get you going in there, especially when, uh, prior to the HoloLens 2. And just some of the infrastructure wasn’t quite where it is today.

Prashant Mahajan (08:03):
Absolutely. Absolutely.

Jeremy Nelson (08:05):
And so I know there was one project you worked on with some faculty in our college of engineering and working with students. Could you explain a little bit more about that and what, what the goal of that was and what it looked like

Prashant Mahajan (08:18):
Once I got access to the, the HoloLenses and I started understanding more about what this device can do. Uh, we had to approach, uh, the school of engineering to see if there were students who would be interested, you know, this is exactly what an academic institution would do, right? Because there are students who are interested in this space who now know more about the technology, which includes, you know, uh, writing a code for this, or designing the user experience, but do not have necessarily access to the domain to which this could be applied. Right? So, uh, I approached to the seniors. So who did this capstone type of a course with me when I went and told them that this is how my world looks like. And so I introduced them to my world and then starting from the when the patient arrives to when the patient is discharged.

Prashant Mahajan (09:14):
And then they had, um, uh, identified certain areas where they felt that they could, you know, uh, use the technology to assist me. So for example, it could be something as simple as the registration using the technology to register patients, to, uh, take pictures and provide diagnosis or to integrate some of the processes or things that I do in the electronic health record, you know, various such aspects. And then we did this for two such semesters and a lot of intriguing ideas came out. Many were not necessarily, uh, have led to fruition, but also allowed me to understand what all can be done, right? So the, the limitations of the project and also the future potential of the project. So that was the value of those, uh, those connections.

Jeremy Nelson (10:05):
Yeah, no. That’s, that’s a great example. I think it’s in what we can bring here at Michigan in terms of experience for students having real world opportunities to explore cutting edge technology, but really make a difference in how that technology gets applied, at a, world-class medical institution. So what was, I, I know there was some talk of, uh, you know, potentially working with a rural health system in the the upper lower peninsula. Could you talk a little bit about that project or what the goal of that was and how that technology might evolve?

Prashant Mahajan (10:40):
Yeah, absolutely. So one of the things, as I was mentioning earlier, I was, I’ve always been intrigued, you know, how this technology can assist the clinician in decision-making. Uh, and what we had actually proposed to Microsoft is say, for example, if a patient were to arrive at a remote, uh, emergency department, say in some rural area, you know, and then the clinician was not, uh, experienced or was probably uncomfortable, you know, in managing that patient provide real time care. Right. Uh, and the reason why we chose this remote area or a rural E.D., Uh, was because we wanted to see, okay, what are the logistical issues that we need to think about, right. About what sort of broadband access they have? Are there any firewalls that are within the institution, you know, are there interrelated or confidentiality related issues that we need to look at?

Prashant Mahajan (11:35):
Do we need to redact images, for instance, you know, again, in real time, but the most important thing Jeremy and I think this is really important to think about is, is this technology and this concept better than the current state of affairs, right? First of all, that unproven, and the second thing is, does it actually even cause harm, right? See if everything looks good on paper or on this video, right. Looking at it, whether it is actually hurting someone. And the last thing we would want to do is to, you know, use technology to harm someone’s safety. Right. So that’s what were looking at,

Jeremy Nelson (12:14):
Yeah, I know that that aligns well with what we’re looking at in the XR Initiative, at least in terms of how do we bring the technology to, to education? Does it, does it help students learn better? Is it, you know, is this technology, where does this technology better than a traditional method or video or, or audio or, and, you know, is it, is it getting in the way, right. Absolutely. Would be a little bit early. So how far along did that, that project get, are you still in process with that? What, any interesting findings?

Prashant Mahajan (12:44):
Yeah, so, so that project is still a little bit slower. Uh, part of the reasons were that we needed to figure out this whole aspect of broadband communication, you know, uh, and whether we can actually transmit videos and audio in real time. Uh, and that is extremely challenging. Uh, but also in the meantime, a very interesting development was happening is that the HoloLens first version was actually being sunset and, uh, new HoloLens or HoloLens 2 was coming up. Um, which had many of where Microsoft, I believe, had addressed many of the shortcomings of the first version. For example, it was now much more comfortable to use sure. We tried to flip up visor. So it doesn’t appear always as a scary device, you know, a lot. Right. And especially in my world, I’m looking at, uh, children who are in suffering from some sort of an emergency medical condition, and they’re already scared. So you just don’t work with them to them. It’s something.

Prashant Mahajan (13:48):
Yeah. So that was one thing, uh, that was one of the reasons why it is a little bit slow, but the most important thing, Jeremy and which I think it’s an important thing, even for the University, right. We need to have a cadre of people which are multidisciplinary, you know, like, so designers, developers, domain experts, you know, people who work with the electronic health records, you know, legal regulatory folks and compliance folks together to even build a very small and a narrow scope of project. Uh, so it is understandable even though, you know, like any other emergency physician, I want to get things done, but I had to take a step back because it is, we needed to address all that. So to answer your question, that project a is probably like a year or two away. Uh, but in the meantime, we are building, uh, various tools and setting the infrastructure in place. You know, we are addressing this whole issue that if I were to take a HoloLens to a different site, what do I need to do to get the system logged on seamlessly? Right. And then start transmitting, wherever I need to broadcast that information. And then how do I even train train providers right. To wear it? What is the onboarding look like? Right.

Jeremy Nelson (15:00):
Yup. Yup. Yeah, yeah, yeah. It’s not a common device there, you know, it’s a running a windows operating system, so you have to log in and there’s protocols you want to put in place, you know, the battery may not last that long. So you want to make sure you address all those things and even now, right. Like how to make sure that the cleaning protocols and how you’re wiping it down and making sure if it’s a shared device absolutely. Transmitting any virus or anything. Right. Okay. That’s fascinating. We’re, we’ve been talking about some similar projects with other, uh, physicians at Michigan Medicine about how to, how could we use the HoloLens for a similar process in med student education and doing rounds. And is there a way to kind of provide that remote support where we may not want as many people in the room as prior. And so how do we continue to teach and provide those experiences for students, um, when, when we have under these new regulations or new constraints,

Prashant Mahajan (15:58):
Right. Then the other interesting thing that has happened, which I believe personally believe is a, is an opportunity too, that we need to leverage, right. This whole pandemic that has happened. It is an unfortunate, uh, instance right, where the entire globe is suffering from an infectious agent, but it is also an important opportunity for us to rethink how we can deliver care. Right. And I think, uh, this entire concept of mixed reality, spatial computing and ambient computing, right, uh, has a, has an important role to play. But again, you have to be a little bit, uh, even though we have to be nimble and fairly fast, you also have to be very thoughtful to make sure that we are deploying, uh, the technology for its most benefit.

Jeremy Nelson (16:45):
Right. Right. I agree. I agree completely well, that’s not the only project you’re working on. I know you have a new project you’re working on and you’re working with a software development firm out of Grand Rapids. Could you talk a little bit more about that project and where that’s going

Prashant Mahajan (17:01):
Well, yeah, that, that it is, it is one of the most exciting projects Jeremy I have undertaken. And I can’t tell you the enthusiasm about the project, right? So initially I was mentioning how we can assist clinical decision making. And this project briefly is to do exactly that. So for instance, I can just lay out the scenario for you, right. So just imagine if there is a, uh, a provider, uh, and I just to a paraphrase that I think the provider is not necessarily always a physician, you know, healthcare providers, the teams are nurses, respiratory therapists. They are all together as a team of providers. That’s how I look at that. And the scenario we are sorting is sort of working on is a contrived scenario where a small child, you know, is just a, you know, it was found submerged in a pool. And the police actually rushed in with that patient, uh, into this emergency room.

Prashant Mahajan (17:57):
And this is a remote emergency room where the emergency room physician is trained to take care of emergencies. But in this instance has not had an opportunity to resuscitate a young child, you know, for a few years assist, uh, that situation or the clinician or the team in that instance. And then the, we are trying to do this is in a very method, methodical, deliberate manner where we are going to prove this in a scientific manner. So what we want to do is we are going to conduct a study where we are going to take this high-fidelity mannequin, uh, into the setting. And, uh, individuals will be randomized, you know, to this metered situation where they can use the current, um, source of cognitive aids that they have, you know, which could be a handheld card, or it could be something on the phone or their computer, et cetera.

Prashant Mahajan (18:51):
Okay. They could be randomized to this software module that we are developing, which currently is going to be deployed through the HoloLens. Right. And then they can choose a, and then we are going to try to find out which, in which of these simulated situations. So for example, using the regular source of information versus a, which could be something in their head itself, right. They know how to manage a patient or use a cognitive aid, this software module that we are delivering through the HoloLens, which one is better. And then the way we are trying to do in this simulated setting for this particular case scenario is how soon did you recognize that? What was the patient’s weight? Did you make any mistake in giving the medications? Because in pediatrics, you have to give it based on the weight. Did you, uh, identify the cardiac rhythm in time?

Prashant Mahajan (19:40):
Did you give the right medications to revive the patient? You know, and all this we can do with the simulator, because we obviously cannot do this on a real patient, doing this on a simulated setting is then we can reproduce, and then we can actually test a large cohort of individuals so that we have enough sample size to actually prove that a and it could happen either is right. Whether the software delivered by HoloLens, is it useful? Is it better? Is it harmful or does it actually worsen the patient’s outcome? So that is the first thing that we are trying to do, but we are also going to measure how much cognitive load, you know, it offloads off you because how much stress to the provider and the team is reduced by the use of this sort of an approach.

Jeremy Nelson (20:27):
Yeah. Interesting. Yeah. That’s a, that’s a good way to look at it. So when you say, is it better is one of the primary metrics, uh, time, uh, like how fast are they getting to these, decisions.

Prashant Mahajan (20:41):
Yeah. Sorry. I think that I’ve been looking at multiple metrics, you know, one which is some them are really objective metrics. Like how soon did you diagnose that you need to give this medication? Or how soon did you, uh, provide the intervention, right. Like for instance, epinephrin how soon did you provide, did you provide the right dose for that individual? So these are quantitative time bound, well, standardized measures. But the second measure that we are looking at, which is a little bit on the qualitative aspect is, does this lead to more stress, right? In the sense, even though the situation is stressful already with the patient, you know, require use of the device and trying to access this information in the device. Right. Does that add to your stress and does it delay the time to, you know, the intervention? So, yeah.

Jeremy Nelson (21:33):
And the stress of the provider is what you’re measuring, not necessarily the stress of the patient.

Prashant Mahajan (21:37):
Absolutely. Here we are. Yeah. The stress of the provider and another important thing too. I just want to highlight here that, uh, there is this whole concept about how the providers, when they work under stressful conditions again and again, and again, then their health suffers. So there’s this whole aspect of wellness and burnout, but it has important implications on patient safety. So who’s your provider, who’s quote unquote, burnt out, right. And then it will increase chance of making a mistake and then the patient outcome is harmed harmed. And no one really studies that in detail. And we probably have an opportunity now to do this in a simulated setting.

Jeremy Nelson (22:17):
Yeah, yeah, no, that’s, that’s a, that’s a big challenge you’re trying to solve and I appreciate your work in that space. I know it’s very important. Um, this is really exciting work that you’re doing really groundbreaking. I think here, uh, one of the reasons I was interested in, you know, the, the XR Initiative was just all the diverse set of, uh, examples and, and opportunities to use XR and in particular in medicine, because my healthcare background, in terms of the future, what sort of concerns do you have about the technology and how we use it or risks?

Prashant Mahajan (22:54):
Yeah. So, so that is, I think, I think this, uh, Jeremy, you, you’re asking a very important question. Uh, and I think, uh, one thing I had briefly alluded to earlier, you know, if you were to go to any one of these technology conferences or conventions, uh, it is very easy to get enamored by a two minute video. Right, right. That can give you an idea of what all this can do, but there is a huge leap from that to actually what is required to make that sort of, uh, you know, uh, technology available because it requires people to be brought together. Uh, it requires the ability to support this multidisciplinary team, you know? Um, so what I think, uh, one of the risks is, and it’s not necessarily a risk, but I think we need to do this in small, meaningful bites that we show that okay, for a very narrow question, can we answer it by using technology, whatever that it may be.

Prashant Mahajan (24:00):
And then the second question you have to ask at the same time is, is it better than the current status state right. And cost or the risks that it adds for instance, you know, worth the benefit that it provides. So we have to be very deliberate because it is easy for me, Jeremy, to, you know, create a cute video and post it up and attract future funders. But if we don’t take the first step of showing that it is not harming patients, that is very important. And I think that’s where I think the risk may be is people may be moving too fast in this space.

Jeremy Nelson (24:42):
No, that’s, that’s really important. And I’m, I’m glad you’re thinking about it that way. It’s yes. When you talk to vendors or see the videos, it’s going to change the world. Everything’s great. And it’s so easy. Well, as you well know when you get into it, it’s, it’s not that easy and it’s not that easy to scale it. Well, what do you, you know, you know, some of what we’re doing with the XR Initiative, you know, what do you want to see the University of Michigan do using this technology to enhance the education of students and faculty? What would you like to see us do?

Prashant Mahajan (25:10):
Yeah. So first of all, I am very, very happy, excited that the University as a whole is looking at this, right. Uh, and they are doing it in a very thoughtful manner. So they have built this infrastructure, obviously Jeremy, you are one of the most important parts of that infrastructure, because now there is a person and there is a series of, uh, individuals and initial initiatives who are looking at it from a global perspective, right? For instance, something can be happening in the sports arena right. To have a benefit or applications in healthcare, right. Or in the school of architecture that can have benefits in the sports arena. So I am very happy that there is this sort of an infrastructure that is being built. So that is one thing that is really exciting to me. Two, I think what could happen is, uh, we to need to have a cadre of developers and designers, you know, which the future pipeline, which could come from the University schools, right.

Prashant Mahajan (26:07):
Various schools, but around an XR lab, which is I think, uh, uh, something that you are leading, but we could feel in the future consider these sort of small teams deployed in areas that are working in this space. Right. So for instance, if there is a team in surgery or is the team in architecture, right? So that is something that we can build. And the third is a forum, uh, which is exactly what I think, you know, your, one of your initiatives that people with different subspecialties can talk to each other and learn from each other. Um, and finally, I think, you know, the future would really be, which I’m very much interested in is if there could be a center or a, uh, a program which we have in the emergency room you know, which is the research and education in acute care using technology or the REACT Initiative that REACT can be expanded, you know, at a much broader level, right? Because then we can get the resources we can apply for funding. We can actually get faculty in the future. Multi-disciplinary trained faculty.

Jeremy Nelson (27:14):
I love it. That sounds great. That that aligns with what, uh, how I’ve been thinking about it and where I think we can go as well. So this has been a great conversation. I’m really excited about the work you’ve been doing and laying the foundation and that you’re continuing to do. And thank you so much for, for thinking about that and blazing the trail and taking the time to share your stories with us today.

Prashant Mahajan (27:38):
No, thank you so much, Jeremy, and I’m really appreciative that you invited me and looking forward to collaborating more with the University. We can change the world. We really can.

Jeremy Nelson (27:48):
I agree. Thank you.

Jeremy Nelson (27:50):
Thank you. Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at ai.umich.edu/xr.