Jeremy Nelson, Director of XR Initiative 

In this week’s MiXR Studios podcast, we talk with Thomas Schwarz, an associate professor of physics at the University of Michigan’s College of Literature, Science, and the Arts. Professor Schwarz is an experimental particle physicist who works with the Large Hadron Collider at CERN in Switzerland. He began exploring VR as a medium to give virtual tours of the Large Hadron Collider including 360 images and video. He and some students were able to use tools such as Sketchfab and the Oculus Quest to bring the Collider to funders and administrators in ways never before possible.

The interest for VR to simulate real experiments was evident to Professor Schwarz early on in his work, and he thinks about how we can explore small or large concepts in new ways. We discuss how the current methods to teach key concepts don’t fully reflect how the real world operates and how this technology can bring new levels of understanding and insight that can’t be accomplished in 2D space. The opportunity to create simulations that vary key elements of uncertainty or changing gravity can easily be done in VR and allow us to add to the pedagogy around measurement and experimentation.

Professor Schwarz was the recipient of the first XR Innovation Fund for his XR Physics Lab proposal. The XR Initiative team has been working closely with his team to create a VR physics lab platform that will allow faculty to create experiments that will replicate the current Physics 141 on campus. The initial project is to create the Projectile Motion lab in VR to allow students to understand the concepts and take measurements. We have been working to create a set of learning goals and objectives framework for each experiment and a storyline that will carry through on each of the subsequent experiments.

The current remote teaching environment is making this work even more salient. We discussed the challenges of VR and requiring students to use this technology when it may not be accessible or a student may get nauseous. The XR Initiative team is also working to create a desktop version of the project to ensure more accessibility. The XR Physics Lab project will include multi-user functionality, and we talk about the challenges and concerns that these types of interactions have on students. Multi-user can create unique affordances, especially for teamwork, and while students are remote. However, there are also risks around dynamics, harassment, and equity in these virtual environments.

We wrap up our discussion with a vision for the future where the platforms and technologies advance to a point where more faculty can create XR content without having to rely on a software developer. The technology holds great promise to explore difficult concepts within education and in particular electromagnetic fields and electromagnetism. When we can get to the point that XR technologies can be as easy to create as PowerPoint slides, we will be able to truly revolutionize how education is delivered.

I always enjoy talking with Professor Schwarz and hearing about how he thinks about Physics and what XR can do to change the way we teach. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 15

Jeremy Nelson (00:13):
Hello, I’m Jeremy Nelson. And today we are talking with Tom Schwarz, who’s an associate professor of physics at the University of Michigan’s college of Literature Science and the Arts. We are talking about his work with VR and the Large Hadron Collider at CERN, and how that led to his project with us to reimagine physics labs in VR at Michigan coming up next in our mixr podcast.

Jeremy Nelson (00:57):
Welcome Tom. Thank you for joining us today.

Tom Schwarz (01:00):
Well, thank you, Jeremy. It’s great to be here.

Jeremy Nelson (01:03):
Yeah. I’m really excited to have you on our podcast and sharing your story and talking a bit about the work we’re going together and yeah. Maybe to get things started. Would you mind sharing a little background about yourself and your journey into XR and what brought us to where we are today?

Tom Schwarz (01:21):
Yeah, sure. So, you know, we started working on XR really VR a few years ago and it was much more closely connected to my own research. This was back in the day when it was just on the cusp of accessibility, meaning you really needed a beefy PC and a headset that was plugged into it and came not prohibitively expensive, or it became accessible in the sense that like, if you had a couple thousand dollars to throw into it, you could do that. And so like I said, it came in through my own research. I’m an experimental particle physicist. I work at the large Hadron Collider at CERN in Switzerland. The large Hadron Collider, that’s basically a big ring of machinery, about 16 miles in circumference. It accelerates protons to the speed of light and then collides them and study the particles that get created when they interact at such high energy.

Tom Schwarz (02:24):
It’s kind of your the same thing that we’ve been doing since the stone age, where you throw a rock and break it apart to see what kind of stuff comes out we’re still doing now. At really high energy what you get to do is as you observe these fundamental building blocks of nature, so-called elementary particles these are particles that are not made of anything else. They’re just fundamental. They’re basically just points with rules associated with them. And, and so those types of experiments involve pretty complicated and I guess what you’d say, abstract science and, and because they’re fundamental physics, they’re pretty dependent on public funding. And so that means outreach is super important. Explain the purpose and the benefits of the work to government administrators and taxpayers. You know, beyond the material being super complicated there’s just flat out physical challenges to the outreach and the experiments in Switzerland. It’s buried 300 feet underground in a cavern with high radiation. So it’s one of these situations where you want to tell the government officials and taxpayers, yeah your money is going to good use, but you can’t see it it’s 300 feet underground in a cavern that’d kill you.

Jeremy Nelson (03:47):
A little hard sell.

Tom Schwarz (03:50):
Yeah. Right, right. That makes it tough. And so that’s kind of where the idea of using VR came in. We want them to be able to give tours, explain the physics associated with the things we’re building and really share something I guess, tangible, but so to speak right with the incredible advancements of the technology that the machine brings. But being able to do it without having to travel to Switzerland and becoming radiation certified and things like that. So yeah, so that was the idea that, and of course you can show pictures and stuff, but there’s no substitute for actually being there and looking up and experiencing it. And, and so to me, VR was the perfect medium for this sort of thing. We, we started putting around with a few ideas in the VR environment there was this, there’s this great group out of University of Chicago that was developing something called Atlas Rift for the Oculus.

Tom Schwarz (04:53):
And that was roughly the direction we wanted to go. But as we’ve chatted before this involved working on the unreal engine, and that really was difficult for us, we were looking for something a little bit more simple. And so we started playing around with, with more accessible platforms and the Oculus Quest and Sketchfab and things like that came along, which were really our Avenue into doing that. And so that’s where we really were able to bring in the scale and scope to the, of the LHC to the public. But in a much more compact and accessible form, it didn’t require us two carts to haul anything anywhere. You know, you just have this little tiny pack that you bring and I’ve brought that everywhere. So once that kinda got going, some students in mind Garrett Merz, who you’ve met, Sabrina Corsetti you haven’t we’ve been working with a high school student, Ella McCauley developed this over the next few years and it’s been great.

Tom Schwarz (05:56):
We’ve used it for many outreach activities, student recruiting. We even brought it to the NSF for a funding review. And it was, it’s been wonderful, really allowing people to experience the experiment. So we have you know, this one case in particular, there was at the NSF, we were asking for asking for money. There was an administrator there, a consultant had been working with us for, for some years and she never seen the experiment and really wanted to right? You want to see the thing that you’re putting so much time into. So we had her put the whole thing on and, and, you know, afterwards she was, was nearly in tears. Being able to finally, you know, see this, this wonderful thing that she’d been putting so much time into. And, and that’s the kind of thing I think that VR brings it’s much, much more closer to actually experiencing the thing that you are trying to, to show and, you know, the project grew from there.

Tom Schwarz (07:02):
So as we realized, this was a bigger platform than just for outreach. We started to think about it for experimental physics education graduate in particular, but, and one of the primary difficulties that well that we run into, really pertinent right now is remote teaching in the sciences. So it’s the big issue here is, is that running real experiments can’t be done remotely, right? That’s a huge problem. And in some aspect, it’s, it’s the value that, that a university adds, onsite university adds is the hands on experience. However there are times when you want to display something or experience something in kind of a more controlled environment or in a case like now where it really needs to be done remotely. And so that’s where this has come in is this idea that with VR, we can simulate an experiment in a near real environment, and that allows students to see the physics as they would in the laboratory.

Tom Schwarz (08:20):
And on top of that, it allows us to expand on the things that we can’t do in the lab, but, you know, like you, you know, that we’re going to do this where we can actually move our experiments to different planets or types of gravity, of course are going to affect things. Or you can dive into the quantum world and see physics that’s really, really small. And that will allow people to do things like you know, every one of us who took a chemistry class is used to just putting little wooden balls on sticks to see the atomic structure and of that is in no way what it really looks like. The beauty of the storm of electrons around protons. And so on far more akin to, to an electrical storm and waves on water, then this rigid structure and only in a virtual world, can you really see this and kind of bring an analogy back to what it’s really like.

Tom Schwarz (09:23):
And so, you know, I, and it has use in, in other ways too, and being able to kind of break things down in the real world, when you do a measurement, there are all these uncertainties, right? So you can’t really control those when you’re actually doing a real experiment. They’re just there. And so that makes it hard to kind of explain in a more systematic approach to the student, whereas in a virtual environment, I can turn an uncertainty on or off. So if I’m throwing darts at a dartboard, I can make it so that the dart hits the bullseye every time, or I can introduce an uncertainty that makes it so that the darts spread all over the place. And the, that carries over into pretty much every aspect of experimental physics. So what we like to say is, is that a measurement, isn’t just a number. It is a number and an uncertainty and the VR environment, allowing us to control that as is an example of a, of kind of value added for pedagogy.

Jeremy Nelson (10:29):
Yeah, no, for sure. And I mean, I love the, the ability you had to, you know, kind of pack up the quest and take it with you and take it on the road. It reminds me of, you know, when I first started in this position, I met with a number of faculty across campus, and I did that. I carried a quest around in my backpack and folks that had never tried it, you know, I’m like, here you go, let’s, let’s go on the space station or let’s go to Mount Everest or somewhere. Cause, you know, it’s, it’s really hard to explain if you’ve never tried it or experienced it. Right? And so until you’re in there, like all the possibilities then open up like, wow, this is, you mean I could do this now? I mean, I could do that? Just like you’re saying with physics, I think the laboratory project that we’re working on, maybe let’s talk a little bit about that now of, so in our initial XR innovation fund, we’re working with your team to, to reimagine, you know, some fundamental physics laboratory course right?

Tom Schwarz (11:24):
Yeah. Yeah. So the first one is, as you know, are absolutely most simple experiments. The idea was we were going to take the most boring experiment out of out of our mechanics laboratory and see if we could make it exciting. So this experiment is basically the, the idea is you launch a puck off of a off an air table, an inclined air table and using the equations of motion, predict where exactly the puck lands based on how much energy you give the puck and the angle. And so it’s just a bunch of students standing around, pulling a puck back on a rubber band and letting them go for two hours. And that really doesn’t sound like the best introduction if you’re trying to excite people into, into physics. So,

Jeremy Nelson (12:18):
I remember doing that experiment in my class

Tom Schwarz (12:22):
Yeah, it can, it’s only fun for so long. So the idea here was, is in a VR environment, we can make that rockets, we can introduce gravity, right? We can do things where introduce gravity from different planets. We can change the mass of the objects. We can create competitions where multiple people can exist in the same environment then launch things at each other. You know, I certainly to, to spice up our experimental laboratory, we could have the students launch pucks at each other and make it better, make it more fun, but.

Jeremy Nelson (13:00):
A lot of releases then.

Tom Schwarz (13:01):
Yeah. It sounds like a liability. So so yeah, so it, it, it’s a, this is the first step in that, and the goal end goal is, is to convert many of these laboratories over into a VR like environment and and take it from there.

Jeremy Nelson (13:23):
Yeah, no, that’s, that’s what excites me about just all the possibilities of building something like this could have for other experiments within physics, within other laboratory courses throughout the university, and being able to expand something more broadly. What sort of support or interest are you seeing from other faculty in physics or other departments? I’ve heard various levels from different folks I’ve interviewed about, Oh, other people in my department don’t really know what’s going on here. And so I’m using this as a way to share.

Tom Schwarz (13:54):
Yeah. So both the chair David Gerdes and Dean of education Tim McKay are quite excited about the project at least as far as I can tell and, and have been very supportive of, of pushing this forward. So now it’s one of these things where we’re putting all this together and, and I think pre COVID we were excited about it and now post COVID, but it’s, it’s, it’s gone from, this would be cool to, wow this would have been really important to have a year ago. Right? This would’ve changed how we did things. And so so I, I think that’s only more the, you know, the, the importance of the project has only been enhanced by, by all of this. It really is too bad that we didn’t start working on it a few years prior because that would’ve

Jeremy Nelson (14:56):
Yeah. Just getting enough headsets, you know, getting that, you know, that’s, that’s a challenge we’re looking at of like, how do we now scale out, you know, access to devices in this hybrid environment, or are people at home? And, you know, there’s a lot of interest right now. Like, Oh, can we do all this in VR for the fall? It’s like, well, not that fast. Right? We have to get more devices and protocols. And did the students buy them? Did they share them? Yeah, there’s a lot to consider.

Tom Schwarz (15:23):
Just a few years away from where I want to be right now. You know people are, have easy access to this stuff, but that’s just not how we’re going. I mean, it’s still, I’m still amazed at what we can do with you know, not too expensive headsets now. It is very, I am very hopeful that a few years from now we will have that situation.

Jeremy Nelson (15:50):
Yeah. And we’ll have the content to support that and a mechanism to scale it, which will be great. This is super exciting in terms of, you know, concerns for the future, you know, does anything concern you about the technology or, or the directions we’re heading?

Tom Schwarz (16:06):
Yeah, a few things. I mean, I do worry about accessibility right? Will people have access to these kinds of headsets, but then even more so that there’s a lot of unknowns with it. And so some folks, when they put on a VR headset, doesn’t exactly make them feel great. Right? They feel seasick, or I can spend, you know, maybe 20 minutes doing it and my daughter seems to be able to live her life in it if she wanted to. But different students will have different reactions to that environment. And so it’s kind of something, I mean, you and I have chatted with this about like trying to make it optional right? Where someone puts on a desktop as well. So they don’t need to experience this in full VR environment. And, and also, you know, augmented reality, is it something that’s more suited to augmented reality?

Tom Schwarz (16:58):
Will that make it a little bit less difficult? And also, you know, there’s all these dynamics where the online community, you know, when, when people work online together it tends to be a little bit more gruff sometimes I think. And so like the social interactions between students, if we make this multiplayer multiplayer environment where people are actually doing experiments as a group, what does the fact that like, people are working remotely, you know, in a virtual environment, how does that change the, how people interact socially? And that seems to be a very big unknown as we’re going forward with things like studio based classes. And in physics we’ve noticed these kinds of difficulties kind of coming up that are just based on different types of social interactions between the students and you know, and how that mixes in with, with pedagogy and how they learn. So

Jeremy Nelson (18:12):
Yeah, I mean, I think it’s, it’s, there’s not a lot of research out there at this point. We’re very interested in exploring this as well and continuing to iterate about where we’re best does a technology work. How do we support that? How do you curate that or build reflection around it or control the environment to some degree you know, there are social VR platforms out there now that just anybody can go into and, you know, the troll culture takes wild takes root quickly. And so we want to ensure that doesn’t happen.

Tom Schwarz (18:44):
Yeah, yeah. It’s super important, right? I mean, if you take five kids and put them around a table that are all face to face, and then you have 20 tables in a room, then faculty and GSIs can kind of walk around and control the discussion and control the environment. Now you do it virtually, right? And that same kind of control or moderation you don’t have. And so do you have GSIs that just pop in once in a while to see how things go? You know, I, how do you create this environment that doesn’t turn into five kids playing Fortnite, yelling at each other. Yeah. I don’t, I don’t know. Right? That’s, that’s a concern.

Jeremy Nelson (19:32):
Yup. Yup. Well, that’s, that’s what we’re going to work on. In terms of the future, you know, what do you want to see us do here at Michigan to enhance the education of students? I mean, you’re already deeply part of it, but what else would you like to see?

Tom Schwarz (19:47):
I mean, in terms of VR?

Jeremy Nelson (19:49):
Yeah using XR technologies, using VR.

Tom Schwarz (19:52):
Yeah. Making them more accessible to the students. I think that would be great. Um and something that you and I have also kind of hoping to do as, as time goes on with this project is to make it so that it’s not always heavily dependent on a developer. So every time we come up with a new laboratory or new idea or something like that having to bring this to a developer it makes it hard. Right? So I, I would like to make it so that we have a platform so that if I am teaching a lecture based class and I want to show the students some idea of something that requires more than me just waving my hands in the air. Can I create that on my own without, without having to run and, and talk to a developer and spend five days, like putting something together. And so I think that’s really, I’d love to see that where I could make it so accessible that if a colleague of mine you know, needs to explain the effects of electromagnetism on charges and wants to make a, a VR demonstration of that, that, that colleague can do that without having to run and talk to a couple of our developers to get that done.

Jeremy Nelson (21:15):
Yeah. No, I love it. That sounds great. What else do you think we should be talking about in this podcast? You know, what other areas would interest you? Are there other folks that you think we should be talking to?

Tom Schwarz (21:30):
Yeah, so the environment is pretty new to me, even though we’ve been working on this for some time, this has been kind of eyeopening to see all of the projects that, that are being discussed in the podcast. So to be honest, every time you’ve put something new out I just got done listening to Sarah Blair’s. I’ve been kind of taken aback by how someone is using tech, using the technology to do something new and different. So I’m not sure I have any advice in that regard as I feel like I’m surprised as how people are using this right now. As time goes on and, you know, you kind of see the breadth of the work. You can start combining these things to do kind of new and exciting things. And I have to admit every once in a while you see something online and or that someone else is doing, and you wonder like, okay, can does that connect to the kind of work that we’re doing as well.

Tom Schwarz (22:31):
As an example I know that you have someone who’s working in nursing, right? To do kind of kind of learn how to do not laboratory work, but learn how to I guess like laboratory work. So it’s kind of a teaching environment where someone is actually getting hands on experience, but through VR, right? And so that connects a little bit to the work that we’re doing, but also a little bit broader when we’re talking about construction and in terms of particle detectors or construction of something new and exciting, can you create a more general environment where someone is actually being able to build something with just general tools that they have at hand, or can you do it so generally or, can you make it so that when you develop something, for example, let’s say right now we’re building particle detectors for an upgrade at the LHC.

Tom Schwarz (23:29):
We bring in a lot of undergraduate students to work on this. And every, you know, undergraduate students that come and they go pretty quick and so you have to retrain them each time. But is this an environment where you can easily whip something up to demonstrate and train how to do some particular jobs? Again, without having to go to a developer, a lot of cool kind of connections, I think with seeing what other people are using this for, to what, you know, you can imagine what we would want to use it for as well. Now, I’m sorry, Jeremy. I don’t have a great.

Jeremy Nelson (24:07):
No, that’s perfect. I mean, that’s

Tom Schwarz (24:10):
I want to keep seeing what other folks are doing.

Jeremy Nelson (24:13):
Well, that’s perfect. I mean, that’s one of my goals is really to share what everyone’s been doing and just allow people to start to make those connections and then come up with ideas like that. So that was, that was perfect. That was exactly what my main goal was. So great. Well, this is great. Thank you for your time and your interest in the work we’re doing together. I find it fascinating and very excited about where we’re going to take this. So I really appreciate it. Thank you, Tom.

Tom Schwarz (24:40):
Yeah, no problem. Me too. This has been, this has been a lot of fun work, Jeremy. Thank you.

Jeremy Nelson (24:44):
Yeah, take care.

Tom Schwarz (24:45):
Yeah, you too.

Jeremy Nelson (24:58):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.