Jeremy Nelson, Director of the XR Initiative

In this week’s MiXR Studios podcast, we talk with Matthew Solomon, associate professor in the Department of Film, Television and Media, about his fascinating work in creating a VR instructional tool for students at the University of Michigan to recreate an iconic scene of Citizen Kane in virtual reality. Students in Introduction to Film, Television, and Media (FTVM), a large foundational course, worked in small teams to create their own version of this scene by manipulating a virtual camera to export video files that they edited together to complete a required course assignment. The project was made possible thanks to grant support from LSA IT.

We talk about Matthew’s vision for this course and how he has been working with the Center for Research on Learning & Teaching’s Foundational Course Initiative. Matthew and FTVM PhD candidate Vincent Longo worked closely with the Emerging Technologies Group at the Duderstadt Center to develop this application in Virtual Reality. Matthew talks about the challenges they faced and some of the ground breaking work they all did together to bring this unique experience to students at the University of Michigan.

In the winter of 2020, more than 160 students were able to engage in this VR tool and had the opportunity to get out of the lecture hall and get hands on experience at the Visualization Studio to work in VR. At the outset of this project, the team wanted to ensure that all students could participate in the assignment, so they created a desktop version of Citizen Kane VR to ensure accessibility. When the university had to move to remote teaching because of COVID-19, the Duderstadt team was able to quickly modify the desktop version to work on laptops and Macbooks. The students were able to create amazing projects and really pushed back against the film to put the Orson Welles character in his place.

We talk about the future of virtual production and what companies like Walt Disney Studios are doing with VR to create the next generation of films and immersive experiences. There are accessibility and cost concerns that could be a barrier to further adoption and we talk about how the work he and others are doing here at Michigan will help pave the way for broader XR adoption. The work that Matthew and others are doing is preparing the next generation of filmmakers and storytellers to shape the future.

I had a great time talking with Matthew about his vision and work and how the students took this application and ran with it to create unique interpretations of this classic film. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 8.

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson. And today we are talking with Matthew Solomon, who is an associate professor of film, television, and media at the University of Michigan’s college of Literature Science and the Arts. We are talking about his transformative work with Citizen Kane VR, a project funded by a grant from LSA technology services and how he brought that to a large foundational course, coming up next in our MiXR podcast.

Jeremy Nelson (00:43):
Welcome Matthew. Nice to talk with you today. How are you?

Matthew Solomon (00:47):
I’m good. Good to speak with you, Jeremy.

Jeremy Nelson (00:50):
Yeah. Glad to spend some time with you and learn a little bit more about the work you’ve been doing and all of the exciting projects and bringing XR to a large class at Michigan. Yeah. Would you mind sharing with our, with our audience, just how you got to this phase, how you imagined it, maybe start by framing what you actually did and then kind of go back.

Matthew Solomon (01:16):
Sure. A year or two ago, I had a meeting with some folks at the Duderstadt Center to ask them, you know, would it be possible to model a scene from an existing film in VR and allow students to kind of experience the action of that scene in a VR environment? So that that’s where it started. And I was fully expecting them to say no way, probably not possible, but instead the reaction I got from Stephanie O’Malley and from others who were at that initial meeting was, sure yeah, let’s do it. Let’s make it happen. And so that’s where it started. And what we found is that we couldn’t model the VR environment using the actual film itself. We actually needed to reenact, remodel the entire scene in order to create it in a VR space. So that is what we did. We chose a scene from Citizen Kane because it’s a film that is very frequently taught.

Matthew Solomon (02:28):
It’s a classic. And we also happen to have the Orson Welles papers here at Michigan. So we were able to get some continuity photographs from the Special Collections Research Center. We were able to use some of that information, some of those rare documents to, to kind of model the space. This has been a collaboration with a PhD student in film, television, and media, Vincent Longo, and he’s done quite a lot of filmmaking. So he was able to actually direct an actor in the Duderstadt studios who basically reenact the three minutes of this scene in a three dimensional space for motion capture technology. Then all that data was, was you know, ported into the, to the space modeled around an avatar who resembles Orson Welles, who’s the main actor in that scene. And the next thing you know, we have this kind of three-dimensional VR rendering of a scene from a classic movie.

Jeremy Nelson (03:35):
That’s super fascinating, super interesting, relevant well, and I don’t think many people know about this Orson Welles collection. Could you talk a little bit more about that and how that even came to be?

Matthew Solomon (03:50):
Yeah, so the, the University of Michigan through the great work of Phil Hallman, one of our librarians, has acquired the papers of a number of major filmmakers. And one of the collections that we have here at Michigan is much of the papers of Orson Welles among other great filmmakers. And so Phil Hallman, and I have been sort of teaching that collection in various ways. But this was an opportunity to use that information, that archival resource for something really dynamic and 21st century. And when the folks at the Duderstadt were then not only able to model the scene, but to introduce a virtual camera into it, then it kind of allowed students not just to be passive spectators of a classic movie scene, but to literally get behind the camera and reshoot it themselves. So that’s been the exciting thing when we piloted it for a class of 150 last term, those students working in groups were able to reshoot the scene each in their own way.

Matthew Solomon (05:04):
And for years and years, we we’ve known how Orson Welles shot it, how he edited, what sound he chose, but those students got to choose for themselves. Here’s how I would shoot it. Here’s how that changes the meaning of the scene. So that was a really exciting piece. I think it’s a real empowering thing to, to put students in the middle of one of the great films and ask them, you know, how would you do it? What would you do differently? So that’s, that’s something that I can’t imagine any other way of doing outside of a kind of VR environment. So again, it went over quite well with the students. I think what they produced was pretty extraordinary and a lot of really interesting variations, most of them wildly different than what we actually see in Citizen Kane. But it became an opportunity for students to analyze you know, not just hypothetically, but to really in a hands on embodied way put to work and actualize what they were learning in the class.

Jeremy Nelson (06:13):
That’s super fascinating. Yeah. I, I sat in on some of those early classes when they met in the lab to, to learn how to access the application and start shooting. So I haven’t heard how things went when, when they finished their product, but you know, what, what surprised you the most? What impressed you? What was the feedback, if you’re able to share that?

Matthew Solomon (06:35):
Yeah, I’m happy to share that. Well we went to a remote environment sort of midway through that assignment and through Dan Fessahazion and his team were able to actually move that tool so that it was accessible on students’ computers remotely, the desktop version of that application, which they built. And what really was extraordinary was kind of the way they were pushing back on the movie itself this, you know, sometimes domineering main character, the way they were able to reshoot it and kind of put him in his place in a way, and the musical choices that they, that they chose really force you to rethink the scene and force you to see it in a different way. And it was really the, the excitement of discovery that, that I witnessed at the Duderstadt center, where students are working together in small groups.

Matthew Solomon (07:42):
And they’re, they’re just really excited about their, each of their respective ideas, how they could do better. There was just so much energy in that room and so much excitement around a black and white movie from 1941. That’s something that that’s kinda hard to achieve in the classroom otherwise, you know, they have a safe distance from, from these old movies, but they, they really took ownership of it. They were able to insert themselves in this movie. And again, the spirit of collaboration was, was pretty incredible, at least that I witnessed you know, they were essentially little movie making teams really thinking critically about the choices they should make, how they wanted to do it differently. And again, it was like energy and engagement that, you know, is hard to achieve in a large class.

Jeremy Nelson (08:39):
Sure, sure. Well, and what is the scene? For folks that haven’t seen our experience.

Matthew Solomon (08:44):
For those that know the film? It’s the temper tantrum scene late in the film when Charles Foster Kane, the main character played by Orson Welles. His second wife leaves him and he responds by kind of trashing her room. It’s a, a scene with just one word of dialogue. It’s mostly within the boundaries of a small space. That was another reason we chose that particular scene. And you know, it’s very dynamic, high energy but it’s this kind of raging avatar. And that’s where students really, I think, thought critically about, for instance, we don’t want to show his face. We don’t, we don’t, we just want to show the collateral damage that this individual has created in this space. And it was just really smart approaches to kind of thinking outside the box not trying to do what’s already been done, but to do something fundamentally different.

Matthew Solomon (09:50):
So that was exciting. And, you know, it was clear to me from observing that, that they learned something they couldn’t have done individually let’s say if they all had been working individually with this app. So that again, I think raises some ideas that I know you and your colleagues are exploring about what’s the possibility for collaboration within a VR space or some kind of hybrid space where some people are in VR, other people are physically present? How can we leverage that to really energize our students and you know, create the next generation hopefully of, of kind of educational tools?

Jeremy Nelson (10:34):
Yeah, no, for sure. And I really liked about this project in particular is, you know, there’s no deterministic outcome. Each student group can create a project that’s different. Every time it will be different, right? In some of the XR VR simulations, you know, you kind of walk through and it’s the same scenario and the same thing happens, you know, you just have to kind of correct yourself and fix it. But in this regard, like you’re really creating something unique every single time.

Matthew Solomon (11:03):
Yeah. No two will ever be the same.

Jeremy Nelson (11:05):
Yeah. So having a tool that can generate that is just super powerful and I think very unique to the program you’ve created and what students can experience here.

Matthew Solomon (11:15):
I mean, it’s part of my kind of educational philosophy of, you know, we’re not here to memorize or parrot back. Something that we’re learning, we’re here to create something new and that, you know, quickly became evident that students were, were empowered by that, you know, that this possibility that really there are very few parameters for this assignment. We just want it to be, you know, roughly the same length in running time as the original scene, but do whatever you like recut it, re shoot it add your own music. So that’s the kind of open-ended assignment that I, that I love and I think also gives our students opportunity even within a strong kind of critical studies analytical course to express themselves creatively and to think critically while also expressing themselves.

Jeremy Nelson (12:17):
Since you did have to move remote, not everybody was able to complete it in VR. Did anybody talk about the differences they were able to experience in VR versus desktop or speed? You know, it’s not a familiar platform, it’s not a familiar method for a number of people, students in particular,

Matthew Solomon (12:37):
What the feedback that we got was that the students really benefited from exploring the scene in VR, you know, exploring every kind of nook and cranny of the room, trying to find and pick their spots, you know, where they wanted to capture it from. But then many of them were more comfortable on a kind of desktop, which is in a sense, maybe a little closer to what you’re doing as a filmmaker where you’re kind of, you know, behind the camera, you’re maybe looking at a video assessed. So I think, you know, as we use this tool in the future we might break it up in that way where some of the exploratory work goes on because many students commented, you know, just imagining what I would do differently didn’t really a substitute for actually being in the environment and being able to see it differently.

Matthew Solomon (13:38):
So I think that to me was also a revelation that you can imagine in your mind how you would do it, but until you’re actually confronted in the space with how things look, that’s where kind of ideas kind of spontaneously combust. So I think as we think about you know, the next time we use this tool, we want to figure out how to best give students time to explore in the VR and then either move back and forth you know, from a, from a desktop or laptop setting where they’re capturing, and then maybe going back into the space and saying, you know, I really could use a shot that is from under the table for instance, or, or whatever it might be. So I think, again, there are possibilities there where some things really do optimize for VR, but then other pieces, you know, maybe work better in other platforms. So I like this idea that, you know, VR could be a piece of our educational approach. It doesn’t have to be all encompassing. And how do we, as educators think about what’s the best possible piece of this assignment that might live in VR? What are the best kind of assignments that might live in VR?

Jeremy Nelson (15:04):
Right, right. It’s a tool, it’s another tool to use in your belt. And, you know, I know a number of production studios are using VR for location scouting. Now they can send out someone to capture that footage and then the director or other folks can explore it in VR and then figure out how they want to shoot it. So that sounds similar in that regard. And you’re starting to see more of that in the industry, you know, the Disney film, the Lion King in VR, and a lot of visual effects and virtual reality for the Mandalorian series. So I, I think it’s, it’s a path that’s coming, right. And it’s great that students here get to be on the cutting edge of that.

Matthew Solomon (15:44):
Definitely. And I think that’s another, you know, of my primary objectives and sort of building this unit into the way that I taught the intro course is this is coming. This is going to be a part of the entertainment industry for many years to come. We don’t know exactly what format or what that’s gonna look like or sound like or feel like, but our students ought to be part of that. And I want them to think broadly about the kind of careers they might take on. If this is something interesting to them, I want to give them some exposure so they can imagine themselves, you know, as an Imagineer or working in this emerging field as part of the kind of next generation of storytellers and audio, visual artists.

Jeremy Nelson (16:39):
That’s great. I mean, I think you’re well on your way in creating something unique here for students.

Matthew Solomon (16:45):
It’s been a lot of fun. I mean, I just have to admit as somebody who has no formal training in VR, it’s just been fun to explore. You know, I’ve seen the excitement in the collaborators from the Duderstadt Center. I’ve fed off that excitement. I’ve fed off the excitement of my, my students. You know, my partners in this, you and others, that’s contagious. And I think we saw that in the intro course, people see this medium. It’s not for everybody, but those that it kinda touches something in them it fires their imagination and there’s a kind of contagiousness to it. And I think if we can bring that to the experience of our students, then we’ve done something extraordinarily positive, even if they decide, you know, this isn’t for them and, and this isn’t something they want to spend a lot of time with. There’s a kind of contagiousness and you know energy, I think that’s coming out of, you know, what you’re doing at AI what’s happening at the Duderstadt center. You know, you just want to be a part of that as a, as a faculty member as a student.

Jeremy Nelson (18:03):
Yeah, no, that’s why I joined the team. Well, how did you, how did you settle on VR? How did that even come into your, your radar as, oh, let’s explore this as an option versus what you had been doing before?

Matthew Solomon (18:16):
Yeah. I think we had been pretty heavily invested in audio visual outcomes for our students where they’re instead of writing critical papers they’re doing audio visual essays in which they make their analytical arguments in audio visual form. And this seemed like kind of the next generation of that to think about how could it actually be different. I’ve been trying to get that, that, that lesson across to students for years, imagine there’s an infinite number of choices. How would things be different if you put the camera differently? If you use different music? If you cut in this place, rather than that place? Every one of those decisions makes a huge difference. So this became the way that we could really imagine alternative possibilities and bring them home and not just kind of discussing it, abstractly, hypothetically, what if, but really putting students in the environment. So that seemed to be something that could only really happen in VR. And to me, it’s a perfect tool really, for this type of learning for this type of exploration.

Jeremy Nelson (19:32):
Yeah. Well, I agree. It was exciting. I’ve tried it it’s very impressive. Well, what other, you know, do you have concerns about the future of XR in teaching? Are there limitations? I know we purposely your team purposely built an accessible desktop version, which turned out to be great for working remotely, but do you have other concerns? Or

Matthew Solomon (19:53):
I guess I, you know, my concerns are, are around issues of accessibility which are kind of two fold. One of which is it takes a certain amount of capital to have the computing power and, you know, the hardware to run these things. So I think that’s going to remain a huge impediment unless we can figure out work arounds. Like the desktop version is one it’s not it’s only partial workaround, but I think as a as a university community, we need to be thinking about how we make these technologies accessible to our students. You know, some of whom don’t even necessarily have reliable internet access, some of whom are borrowing machines for the semester, from IT there’s very real issues of access that we can’t lose sight of in this sort of glorious technological future that VR seems to promise. So I would really, you know, I want to sound that alarm, that every moment that we, we just need to be mindful and we need to make sure that we’re investing our resources in making this widely available.

Matthew Solomon (21:13):
I don’t know what the market penetration is for VR headsets and you know, what the socioeconomics of that are. But we as a university community, we ought to keep an eye on that. And then the other issue is around varying abilities and how VR can be a space where people with both you know, physical disabilities can really function and get the most out of that environment, that experience can we custom build these environments so that they conform to all kinds of accessibility protocols, you know, are there accessibility guidelines that are in place to kind of make sure that VR spaces that we’re creating are fully accessible and compliant? You know, in that way? Again, I’m not part of those discussions, but I, you know, I trust that they’re taking place and that provisions are being made.

Jeremy Nelson (22:13):
Yeah, there’s actually a whole group called XR access. That has a Slack group and some symposiums that are very active in this space in terms of education, content, authoring. I’d be happy to connect you there.

Matthew Solomon (22:27):
Sign me up.

Jeremy Nelson (22:28):
All right, well, I’ll connect you with that group. It’s, it’s great to see what’s happening in that space and participate in and help shape it as well. You’re already helping set some precedent for the education of students here at Michigan, but what do you want to see us do to further enhance a student’s education with XR?

Matthew Solomon (22:45):
Well, I mean, I’ve got specific goals for this, this project that you’re aware of. I think, you know, we ought to think about how we might use this, not that we must, or that it’s the right tool for every application, but, you know, I would encourage my colleagues to, to think about how might we use this? And I think in some sense, it’s our responsibility, at least in my department to be preparing students who can function in this realm, because as you say, there are big pieces of the industry, animation, that seem to be shifting into this space. So we want our students to be graduating with Michigan degrees, where they’re comfortable and prepared to compete for those positions and can help determine what the future of entertainment looks like and not be passive consumers, but really push this field in new directions and bring their talents and what they’ve learned about storytelling and you know other kinds of audio, visual entertainment into the, into the space.

Jeremy Nelson (24:00):
Yeah, well, I, I have that same mission. I couldn’t agree more around helping prepare students to shape the future, as well as be prepared to participate in that, that future. This has been, this has been great, you know, you’ve, you’ve heard a few of our podcasts, you know, what other topics do you think we should explore? Who else should we be talking to? What’s interesting to you and your colleagues?

Matthew Solomon (24:21):
I mean, I’d like to, to get a better sense of the, the student eye view of this. I know there’s a couple of clubs on campus. I sense that there’s a lot of student excitement. Many of them are gamers. I I’d really love to get a better sense of what that perspective is and you know, get a clearer idea of where they’re coming from, how they’re using this medium, who’s using it for one and where they see this sort of playing into their education. Cause my sense is probably they see it as something separate from their education. So that’s something I’d really like to, to query and to know more about you know, besides what’s going out in the entertainment industries, you know, other film schools are picking up VR as a piece of their kind of mission. And I’m curious how that is happening in places like New York and Los Angeles and whether there’s a possibility that, you know, will there be another location or space where VR really starts to take root and you know, move in new directions.

Jeremy Nelson (25:45):
Yeah, well, let’s make it happen. And I’ll definitely start interviewing some of the students and getting their perspective. That’s, that’s a great area to explore. Well, thank you very much, Matthew, this has been fascinating and really exciting love to hear about all the student feedback and what they created. Thank you again.

Matthew Solomon (26:05):
Happy to share. Keep up the good work.

Jeremy Nelson (26:16):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.