Jeremy Nelson, Director of XR Initiative

In this week’s MiXR Studios podcast, we talk with Hera Kim-Berman, a clinical assistant professor in the Department of Orthodontics and Pediatric Dentistry and the program director of graduate orthodontics at the University of Michigan’s School of Dentistry. Hera has been one of the leading faculty at U-M to explore and push the boundaries of VR in teaching and learning. In our discussion, we talk about her work in bringing the Oculus Go headset to more than 100 first-year dental students. We discuss her early work with the Emerging Technologies Group at the Duderstadt Center to experience what is possible and explore what these tools could do in her teaching.

Through a connection with Amy Klinke at the Business Engagement Center, Hera was able to connect with the Oculus Education Group. Hera submitted a proposal to study how dental students learn and visualize dental anatomy. Oculus was supportive and donated roughly 150 Oculus Go headsets, a couple Rifts, PCs, and Quests. She worked to create 3D models of teeth for students to assess their own work by utilizing the Arthea platform that was purchased by U-M.

Each of the 109 students in her class were loaned an Oculus Go headset for the semester that they could use to explore 3D models and evaluate their work. Instead of going to the lab, they could study at home, and they could bring the headsets into class to explore 3D models in a multi-user session with Arthea. Students were excited to explore VR and early findings were that students were more easily able to assess their mistakes by viewing their work scaled up in VR as compared to evaluating with the naked eye.

The XR Initiative awarded Hera Kim-Berman one of the inaugural XR Innovation Fund awards in February for her submission called “Comparison of Student Learning of Head and Neck Anatomy and Diagnosis of Pathology Using XR.” In this project, Hera is looking to explore how students understand 2D cone beam computed tomography (CBCT) versus 3D virtual representation in terms of learning anatomy. Their goal is to see if students can use VR to understand, diagnose, and interpret 3D imaging in a more intuitive way. They are using a platform called ImmersiveTouch to create 3D representations of head and neck from computed tomography (CT) and magnetic resonance (MR) scans.

We conclude our discussion by exploring what challenges and opportunities lie ahead for U-M. Hera discusses the current research and publications around VR in medical and dental specialities. She advocates for the university to continue to invest in XR technologies and services and support faculty that are wanting to bring this technology to their students. She is passionate about exploring the science behind the learning with these XR technologies and how to evolve assessment to create evidence based best practices.

I found the work that Hera Kim-Berman was doing is ground breaking in terms of dental student education and enjoyed our conversation. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 14

Jeremy Nelson (00:10):
I’m Jeremy Nelson. And today we are talking with Hera Kim-Berman, who is a clinical assistant professor in the department of orthodontics and pediatric dentistry and the program director of graduate orthodontics at the University of Michigan School of Dentistry. We are talking about her work in bringing 3D visualization and virtual reality, to classes of over 100 dental students to explore teeth and dental anatomy in VR. Coming up next and our mixer podcast.

Jeremy Nelson (00:54):
Welcome Hera, thank you for joining us today.

Hera Kim-Berman (00:57):
Thank you. Thank you for having me.

Jeremy Nelson (00:59):
Yeah. We’re excited to talk with you and share some of the exciting work you’ve been doing over the last few years. I think it’s pretty fascinating and I’d love to just like, how did you get into XR? Like what, what drove you to this space? You know, how did this start?

Hera Kim-Berman (01:15):
Yeah, it is, it is a very interesting story. I came to the University of Michigan School of Dentistry as a new assistant clinical, clinical assistant faculty here at the school. And, and, but, you know, really my training is in orthodontics. And so finding this virtual reality, it was a little bit unusual. So in 2014, when I came to the university dentistry was very big on 3D printing and fabrication. And so I really reached out to sources within the university and I came across a 3D lab now known as the Duderstadt Center, right. It was part of the library system then, and I went down to the folks there and I just wanted to see what they were doing in 3D fabrication, not knowing that they had this emerging technology group. Then it was under Eric Maslowski Theodore hall and Sean Petty, as well as you know others were there and I got to meet Ted Hall and he was working on this project in VR with Oculus headsets. And he was also working on jugular software, our in house VR software that was developed. So I got very much interested and he showed me MIDEN which is kind of like a Michigan version of cave type of virtual environment. And Ted actually uploaded a skull. And at that moment I had this you know, thought that I could actually utilize this technology in teaching our students, dental students, our residents, to really learn about the craniofacial biology of our patients and and do it in a very intuitive, real immersive way. So that’s what piqued my interest on this. So that, that was the the Genesis of just getting started because prior to that, my experience in VR was just very minimal.

Jeremy Nelson (03:49):
Sure, sure. What did you see when you were in that MIDEN the first time that the light bulb went off or what, what did you see differently than you’d ever experienced in other forms?

Hera Kim-Berman (03:59):
So I was shown a the the living human project, I think where you can actually dissect a a cadaver. And it really looked through sections of this and walk around the body almost like I was in a gross anatomy lab and doing it virtually. And it was actually larger than life in MIDEN. So you can actually kneel down and look underneath and walk around. And it was really a fantastic experience for anyone who has not had that experience. And the, the other thing that really drove home was that they really, they had a a skull there. And I was able to actually look at the skull from all views, including the view from back of the throat and really visualizing that because it’s almost impossible, you know, to get that view and appreciate that view. So for me, that’s really what was exciting and it was again, larger than life. And it was, it was very immersive, real, I lost track of time, really trying to look at everything. Yeah. So I thought that this would be a wonderful way to engage our students.

Jeremy Nelson (05:25):
Nice. That’s great. So then where did you go from there?

Hera Kim-Berman (05:26):
So from there, I kind of collaborated with now the emerging technology group and really worked with Jugular software. And and I thought that maybe, you know, I, we have as dentists tremendous data, three dimensional data of our patients, head and neck area using DICOM files the 3D imaging technology that is out there, CT images. So we can utilize these patients as examples, and then we can actually develop modules for simulation. So orthodontists actually are very much involved with skeletal changes in our patients. So for adult patients that involves orthodontics or braces in addition with corrective jaw surgery. So the first time that I envisioned this tool to be used was in jaw surgery simulation in using virtual reality. Currently they, the standard for teaching jaw surgery is in only one plane, you know, anterior, posterior plane with a two dimensional x-ray of a head film, a cephalometric film.

Hera Kim-Berman (06:54):
And the students are taught still to trace the outlines of this x-ray and then cut out the parts and then move them around. And I don’t know if you guys remember that a game that we used to play called Colorforms where it was just pieces of, of you know, maybe the jaw and then you would place them and stick them on the lower jaw, and then you’d place them and then stick them onto a new position. And that’s essentially what we were doing to, to help students visualize where the jaws might be. So as a clinician, you want good predictive outcomes, right? Our prediction in our simulation should really mimic real situation. And we were really far from that using 2D doing it to 3D looking at only one plane. And so even though, you know so that was very limiting. So I really wanted to incorporate this emerging technology into that to make it a little bit more to increase the fidelity really of the, of the learning tool.

Jeremy Nelson (08:12):
No, it sounds fascinating. It sounds, sounds like there’s a lot of opportunity to, to enhance the education and the way students are taught and experience this, you know, inherently spatial environment.

Hera Kim-Berman (08:28):
Yes, that is correct. Another thing that really helped me within the university is the business engagement office in that they were able to connect our, our faculty with corporates I guess sponsors or corporate partners really. And they were able to connect me to Oculus education group and we have formed a relationship with Oculus so that they were absolutely interested in gifting us some equipment and so forth to really investigate the effects of EVR in dentistry and dental education.

Jeremy Nelson (9:17):
That’s great. So, so what did that look like? So how did that, you know, they connected you and were they interested in something already, or were you already interested in something and together you came up with an idea?

Hera Kim-Berman (10:41):
Yeah, so really it, it really does involve sort of like they were so open to helping with any part of education and being involved. And I, I basically just kind of had a conversation and submitted a proposal as to what I would like to study. And I talked about the jaw simulation, how I wanted to really develop that. And they were like, I think that’s a great idea. Here are some equipment and some computers, so that you guys can work on this. And so it kind of turned out to be a very simple process and just you know, something that they were very much supporting. I would actually kind of check in with them once in a while and kind of update them on what was happening. They also kind of try to to help at the faculty involved with Oculus education group and to invite us to the Oculus connect meetings. So I was able to attend the OC five and OC OC six. So those were some of the the real opportunities where I saw a lot of their products on display and upcoming things. So that’s that was very informative to me to go to these corporate events. It’s a very different, it’s very different than university events.

Hera Kim-Berman (11:04):
It was very interesting.

Jeremy Nelson (11:06):
Well, that’s super exciting. So, so they gave you, what did they end up, what types of headsets, where they the Go, the Quest, the Rifts?

Hera Kim-Berman (11:13):
Yeah. so they were, so when I first started with them, the rift was really the only device that was available, the consumer version six. So those were the original ones. And then they, from there we evolved and with their new equipment coming out, they were able to gift the school of dentistry many hundreds over a hundred, I would say, 150 Go devices.

Jeremy Nelson (11:45):
Wow.

Hera Kim-Berman (11:47):
Uh they were also able to gift a few Quest devices as well. And they have also supported us with some startup funds, as well as with some computer help. So they gave us some laptops that could run the rift headsets. So, you know, without their involvement and their partnership, I don’t think I could do everything because again, the cost of some of these units, especially the processing power, the computers, those are the, the limitations for us.

Jeremy Nelson (12:27):
Yeah. Wow. That sounds amazing. Like a great accelerator. And so you have so many headsets, so what did you request that many headsets? Or like, what was your thinking? What was your request? I guess

Hera Kim-Berman (12:40):
So, yeah. So what, what I wanted to do there was also a a small startup company developed by a lot of umich grads here at the university and it was called it was a a software just a visualization software. And they named that Arthea. And since then Arthea has been acquired by the university college of engineering. And we wanted to use utilize this VR visualization software program using the Go and deploy that and implement that with our incoming class, back then class of 2023, the first year dental students, and incorporate VR into visualization of dental anatomy.

Jeremy Nelson (13:37):
Wow. And how big of a class is that?

Hera Kim-Berman (13:40):
Yeah so our class size consists of 109 students.

Jeremy Nelson (13:44):
Wow.

Hera Kim-Berman (13:46):
Hence the number of headsets that were needed to really try to implement this. And so this class had a combination of looking at real dental anatomy real human teeth, examples of plastic teeth, let’s say. And they’ve also had the advantage of using a 3D imaging software on a desktop or a laptop. And then they also had the advantage of looking at it in VR. So we were, so we were really testing to see whether in VR or 3D and looking at real models where, you know, when does it make a difference? Does it make a difference? Yeah. And we conducted a study and can we test student knowledge just using virtual images? So these were some of the tests in it and things that we were conducting, which Oculus education group supported.

Jeremy Nelson (14:52):
Yeah. No, that’s great. So, so you, did you hand out a device to every student or did they come into a lab and check them out? Like, how did, like, this is, this is the largest implementation of VR at Michigan, for sure up to this point.

Hera Kim-Berman (15:06):
So yeah, each student so we, you know, barcoded each headset and for the first semester and for the first year of the dental school education, the students were allowed a, a loner type of a Go device to take home with them, to utilize, to explore, and also use it for coursework. So they have had it for a full year now.

Jeremy Nelson (15:39):
Okay.

Hera Kim-Berman (14:40):
So that’s how we’re implementing it, so they don’t need to actually go to a lab. And then we would ask them to bring in their headsets for coursework and things like that too.

Jeremy Nelson (15:52):
Nice. Nice. So what, what were your early findings? What were some interesting learnings along this journey? I’m sure it’s lots of, lots of interesting thoughts.

Hera Kim-Berman (16:02):
Yeah. So there’s lots of, lots of interesting things I can tell you that students are very interested in exploring VR. And however, the, the, one of the more difficulties that I have, and it is probably something that a lot of other educators are really looking at is sustaining that interest, you know, like so that was a, that was one thing that I have you know, they, they, I think students need to value this additional use of technology they did not see it as a, an added, you know tool that they just have to use, or they’re forced to use and embracing this technology over a long period of time and then utilizing for their own benefit. That is going to be, I think one of the challenges going forward the use of the VR again, visualization, it’s much more enhanced.

Hera Kim-Berman (17:09):
And so they’re very they have often a common comment that I get from the student feedback from the students is that I didn’t know that, you know, this whatever manipulation that they did to the tooth to prepare the tooth that they had so many mistakes, it’s so difficult to just look at that little thing just looking at it and then looking at it, you know, much larger in a, in a three dimensional, in a virtual environment that they really do see the fine details that were missed. And it acts actually helps them better assess their work. And that’s really my my forte into going into this, my focus into going into this is to really look at how students are assessing their own work using VR, rather than just visually with the naked eye.

Jeremy Nelson (18:15):
That’s great. That’s exciting. So, so, you know, we funded a new project with you as part of our inaugural XR innovation fund from the XR initiative, and we’re pretty excited about it. I’d love to learn a little bit more. What are you hoping to explore? What does this look like? That’s a variation or kind of an evolution of what you’ve been doing?

Hera Kim-Berman (18:39):
Yeah. So even though the use of cone beam computed tomography CBCT that is much similar to CT images, 3D images that was really introduced you know decades ago for the use of dentistry, however, in dental schools the students have very limited experience with use of this 3D imaging. And they don’t really have a really full grasp of how it can help them and teaching it is a little bit of a challenge. So what we decided to do is to go ahead and see how these images can be interpreted using a 2D screen versus a virtual environment. So what I would do is to marry actually anatomy of the head and neck, and marry that with 3D imaging radiographs and CT imaging marry that together. So that students really see and learn anatomy in context of clinical analysis and diagnosis for the purposes of determining pathology and determining a normal anatomy. So that’s really the, the goal is to see whether we can incorporate VR and for students to understand diagnose, interpret these 3D imaging in a more intuitive way.

Jeremy Nelson (20:25):
Nice. So, so what will that look like in terms of the, the student they’ll have the traditional learnings, and then they’ll go into VR to explore, how are you thinking about it? What does it look like?

Hera Kim-Berman (20:37):
Yeah, so we’re thinking about yeah, so there are some traditional learning of what radiographic images might look like. And then they would go to a lab and they would kind of explore an overlay almost and slice through the individual patient’s skull and be able to slice through and look at the images in concert with the anatomy very much like my experience when I first came here in my day, looking at a stall and looking at all different aspects, but now imagine just overlaying an X Ray, right, of that particular patient. Identifying pathology tumors and cysts, or just really identifying normal anatomy. And I think that that would be, I think you know, something that would be benefiting the students engage them as well in learning their what they have to to learn in order to become good clinicians.

Jeremy Nelson (21:47):
Yeah. No, it sounds, sounds fascinating. I remember we tried this application when, when I first met you and it was, it was fascinating to me like, Oh my gosh, this is like a whole, it’s a whole new world. And it was super exciting to me. I mean, in terms of the future, do you have concerns about using XR for teaching and learning and perhaps what do you want to see Michigan do in this space? Furthering the type of work you’re doing?

Hera Kim-Berman (22:12):
I think that you know, still, I mean, I think that a lot of the dental faculty are unfamiliar with VR, and I know that there is a lot of of the of the scholarship coming from from the VR side into medicine and into into dentistry right now. So I think that, you know, we should support the university should support funding and initiatives like that from AI to make sure that this science happens and then it becomes disseminated so that everybody understands that it can be a valuable tool. Well, I would like to see more of that coming from the university. And I think that in general, if you look at the if you do a pub med literature search from 2019 on you see this this really huge increase in VR and medical and dental literature coming out. But I would like to see a little bit more you know science and a little bit more assessment and, and really evidence-based. And I want really from all of that, maybe best practices for using VR in clinical situations. That’s what I hope.

Jeremy Nelson (23:41):
Yeah, no, I agree. I agree. I think it’s, it’s a, it’s a powerful technology and we just have to keep figuring out how best to deploy it or where best can it help in the educational processes. You know, sometimes it’s better and sometimes it might not be, yeah, no, this is, this is exciting work. I, I love the journey of that you’ve gone through in terms of going in and seeing it for yourself and then taking that back and then bring it into a very large course and continuing that process forward. This is, this is very exciting. Are there other topics you think we should be exploring in this podcast or are there other folks that we should be talking with here at Michigan? What do you think?

Hera Kim-Berman (24:23):
Yeah, I you know what, I would really love to see the perspective from this the emerging technologies group, you know, specifically those people who have contributed to developing the Jugular software. I think that’s really interesting because it’s uniquely University of Michigan, right. This, this software. So I would be interested in seeing them and hearing about their story.

Jeremy Nelson (24:54):
Yeah, no, for sure. Yes, we were, they’re on our list, so great. Yes, that’s great. Well, thank you so much. This has been great. It’s fascinating. We’re excited to continue the work and thank you very much.

Hera Kim-Berman (25:08):
Yes. Thank you, Jeremy. And thank you for all the work that you are doing and supporting faculty research and you know, student experience here at the university.

Jeremy Nelson (25:18):
My pleasure.

Jeremy Nelson (25:30):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale, that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.