Jeremy Nelson, Director of the XR Initiative

In this week’s MiXR Studios podcast, we talk with Jonathan Rule, clinical assistant professor of architecture and urban planning, about how he is shaping the future of architecture education through augmented and virtual reality. Jonathan teaches courses in construction and architectural design and has been exploring technologies such as Google Cardboard and Unity to immersively experience architectural designs.

We talk about Jonathan’s previous experience with immersive technologies and his new XR Innovation Fund project, Augmented Tectonics. He is looking to enhance his course, which is offered undergraduate and graduate students, so he is able to teach more than 100 students about construction design. We are working on a multi-user virtual reality experience to explore construction materials through example structures and challenges.

The Taubman College of Architecture and Urban Planning is exploring XR technologies more broadly to bring the 3-D models that students create into augmented reality on the Microsoft HoloLens with tools like Fologram. We discuss some of the challenges with creating XR experiences and access to XR equipment beyond labs on campus. There are opportunities to collaborate across the many schools and colleges at the University to help solve these challenges and give students the best opportunities to shape their future.

I had a great time talking with Jonathan about his thinking and the new Augmented Tectonics projects we are working on together. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 9

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson. And today we are talking with Jonathan Rule, who is a clinical assistant professor of architecture and urban planning at the University of Michigan’s A Alfred Taubman College of Architecture and Urban Planning. We’re talking about his work with augmented and virtual technologies for architecture, his new XR innovation project, Augmented Tectonics. Coming up next in our mixr podcast.

Jeremy Nelson (00:38):
Hello, Jonathan. Welcome.

Jonathan Rule (00:40):
Hi Jeremy. Thanks for having me.

Jeremy Nelson (00:42):
Yeah, I’m glad to have you on this week’s episode of mixr podcast and looking forward to learning more about all the exciting work you’ve been doing the innovation fund project that we’re working on together, and yeah, I’d love to hear how you got into this. Like what brought you to this space, maybe some of your projects and your, your experience teaching in XR.

Jonathan Rule (01:04):
Totally. yeah, so I would say like, I’m sort of new to the XR world in a way and, and I’ll kind of unpack that and explain that a little bit. So I mean, I’m an, I’m an architect by profession. And I teach at the school of architecture and urban planning where we focus, I think mainly on ideas of fabrication and representation, in architecture and in our discipline. So architecture is, is focused more on the tangible aspects of the built world. And so this is something I think that architects have seen and they’ve, they’ve, they’ve been kind of observing it and seeing how it fits into how they work and how they design the built environment. And so it’s something for me that I’ve, I’ve kind of kept in the back of my head and I’ve seen it kind of coming into being, and I’ve, I’ve been more and more interested in it as, as time has gone on.

Jonathan Rule (01:56):
I can sort of maybe kind of go back a little bit further in my history of my experience with, with extended realities and how I’ve seen it used in architecture. Going a little bit back to my undergraduate years when I was a research assistant at the university of Buffalo under Shahin Vassigh, who’s now at Florida international university, FIU, and she was developing a digital textbook for the visualization visualization of structures, titled interactive structures visualizing structural behaviors. And so I think that work again, it wasn’t necessarily at XR, but it was during that first wave where eBooks and digital books and things like that started coming into play and coming into the classroom. And how do you bring technology into the classroom? And even at that sort of onset, we were already discussing like cave environments and how you could begin to bring these 3D models that we, we knew how to build as architects into an embodied reality that students and, and professionals could begin to interact with and really kind of unpack again like a lot of the constraints that we have or our space constraints.

Jonathan Rule (03:10):
And so virtual environments, all of a sudden become kind of this new frontier where, where we’re, we’re kind of those, those go away, because you, you, you kind of have these new virtual environments that you’re interacting with. So I’d say that’s sort of like the start of like, where I kind of came from. And so that has sort of again, developed through how I teach and how I research and understanding of materials and methods of construction and sort of, kind of going back to that that original work or that work that I was working on in undergrad and beginning to couple that with some of my newer work and research and, and trying to bring that also again into the classroom which is where this, this new project that I am currently working on with you guys at Academic Innovation begins to start.

Jonathan Rule (04:00):
And so I think like for me, there’s really two ways to begin to think about, about this technology. One way is again, a pedagogical approach where we’re looking at it as a didactic tool that can convey certain concepts and which I’m exploring through the augmented realities project. And then there’s the more open ended or exploratory design tool that can help us see things differently or think about spaces differently. So just kind of to give a very brief example of this and how it might begin to interact with ideas of teaching at the university. We have a program at Taubman college, which is the arch prep program which is it’s in conjunction with the Detroit public school system where we have juniors in high school that are taking courses in architecture. And so they always invite faculty to do workshops like one day workshops or two day workshops with the students.

Jonathan Rule (04:54):
And so last year we had the opportunity to, to work on one that, that began to leverage ideas of XR for spatial thinking and adaptation where we were actually going in with a three 60 imaging photo Photoshop editing, and then using Google cardboard to actually immerse students in these alternative public spaces for the city of Detroit. So I think again, yeah, there’s a lot of cool opportunities in this, both as a teaching and as sort of a technical tool, but also as an exploratory in getting both young and old excited about it.

Jeremy Nelson (05:27):
Yeah, for sure. That’s, that’s super fascinating. How did the students react to that? In terms of, you know, just the technology, the imagining the space, what were their reactions?

Jonathan Rule (05:38):
Yeah, I mean, they loved it. They really enjoyed it. It was, it was kind of at the beginning, we, there was a little bit of a learning curve, like trying to make sure all the computers were working properly and they had enough power and all that to do that.

Jonathan Rule (05:50):
But as soon as kind of, we got over some of those hurdles and students were really able to kind of let loose and explore it was kind of, they didn’t quite understand it at first cause they were there manipulating images in sort of a two dimensional space on their screen. And then all of a sudden, as soon as we, we flipped it to their phones and Google cardboard, and they had had that to experience that kind of blew their minds a little bit. They’re, they’re, they’re very excited with the final outcome outcome of it.

Jeremy Nelson (06:18):
Sure. I could imagine, I could imagine. And so you’ve been teaching with VR this last year, is that correct?

Jonathan Rule (06:26):
Yes. So I did launch a an experimental seminar, which is giving architectural students sort of like the insight basic insight into how to begin to develop environments using the unity software. So coupling softwares that they already know Rhino 3d and then how do you begin to model and bring things into unity and then also begin to develop interactivity within those environments? So not just using it as a passive visualization tool, but how do you begin to actively interact with objects within that virtual space?

Jeremy Nelson (07:01):
Hmm. And what do those affordances give you differently than how you’ve taught something like this in the past?

Jonathan Rule (07:07):
Well, I haven’t taught it in the past, so I learned it in the past. This is sort of a, kind of a new branch out to, to kind of explore, I think, what are the possibilities of new technologies within again, the discipline of architecture? I think it’s something that’s relatively new and so we’re not sure exactly what direction it should go in. And so this, this is more of a way of exploring what possibilities there are and how we might advance it um more specifically for the architecture engineering and construction industries. And so I think that’s what the exploration initially of the seminar was. Really trying to sort of unpack this, this new tool that we have at at our disposal and how that can that can develop into something that’s sort of beyond the repertoire of tools that we’ve currently been using within architecture and how that can kind of help us as we design or as we build our structures.

Jeremy Nelson (08:08):
Cool. Cool. Well, what surprised you the most about what the students were able to do or what they learned in the seminar?

Jonathan Rule (08:14):
Well, I mean, unfortunately sort of, because of the situation that we’re currently in, we, we had to cut it off mid-stream. And so there was a lot of work that we were doing going towards VR and we had to quickly adapt it to use a slightly different platform and approach. So we really didn’t, I think get to the end goal that we wanted to with it, but it was kind of amazing to sort of see the students all of a sudden, begin to pivot as quickly as they had to and go from something where they were working primarily in VR, this embodied experience, but then also be able to kind of still develop some ideas about interactivity and how you might begin to leverage this, to explain the sample buildings that they were developing through more of a traditional first person kind of onscreen view of, of the environments.

Jeremy Nelson (09:06):
A lot of things have changed and we’ve had to adapt quickly, which I think leads us into, you know, the, the XR innovation fund project we’re working on augmented tectonics. And we’re excited. We’re excited about that. We’re excited to work with, with you and your team. We’ve already had some early meetings and it’s exciting where the direction is going. So, you know, what, what’s your goal with that? The vision. How do you see this impacting your work broadly in Taubman college?

Jonathan Rule (09:31):
Sure. Yeah. So so this project is building on a project that I was previously working on through a grant for our CRLT at the University of Michigan. And the idea for it was we, we teach this course, which is both for juniors in the undergraduate program, as well as three year graduate students. And so these are students that they don’t have the graduate students don’t have an architecture background necessarily, and the undergraduate students, they’re, they’re taking it for the first time. And this is, this is their construction course. So this is where they, they begin to understand materials and processes and the translation of three dimensional ideas into two dimensional representations or two dimensional drawings. And it’s a lot, there’s a lot of information that, that’s sort of thrown at them. And one of the biggest issues that we found that we have run into with this course is that it’s very difficult for the students to imagine a lot of these things, if they haven’t experienced it or seen it in the real world.

Jonathan Rule (10:37):
And the biggest issue is that it’s difficult to get a hundred, some odd students onto a construction site for obvious liability reasons and so forth. And so every single time we have like a construction site on campus, we try to get them in. But it doesn’t always work out. And so this, this idea kind of stemmed from that where we said, okay, well we have also the opportunity to maybe simulate this simulate these, these site visits and allow students to experience or embody these these construction sites. And I think it goes beyond that too. It’s not just sort of like walking around a building and trying to understand that, but it’s also I think making the, the subject more interesting in a way or trying to to get the students a little bit more excited about it beyond the, the standard lecture and lab that we currently have.

Jonathan Rule (11:34):
I mean, you could probably imagine me talking about concrete for two hours, the attention span of somebody will probably drop off in 20 minutes. So we’re hoping that, that through this and through sort of this new way of looking at learning this material using augmented and virtual realities that there will be a greater retention and a greater excitement over this particular subject matter within, within the architectural program. And so that’s sort of where, where it stemmed from, and that’s, that’s what we’re, we’re hoping to achieve through its development through, through different material logics and so forth.

Jeremy Nelson (12:12):
Yeah. It’s super exciting. Just some of the early designs and work your team’s bringing to the project we’re very excited about. I think it’s going to be an exciting opportunity for the students and hopefully for you and other instructors working in this space. Are there examples of XR that you’ve experienced, I know there’s, there’s other work at Taubman college or across the university, that represent good examples of teaching with this technology in your mind?

Jonathan Rule (12:39):
There’s quite a few. I think namely though, there’s, there’s one that a number of us have been working with which is more maybe again, trying to dovetail this fabrication and, and the new kind of digital space or virtual space that are augmented realities that people are working with. And so like with Matias del Campo is another professor at the school of architecture has been working with it a lot lately. I’ve been working on it as well, too, with some of some students but it’s a software called Fologram, which is, is quite interesting in that it links up directly with the software that the students usually are using for their, for their modeling and drawing and their design studios and what it allows you to do. And so this, this is using either your phone or a hololens where they can begin to model things in real time, and then actually see the projected augmented image update in real time through the device.

Jonathan Rule (13:42):
And, and well, it could be updated in real time or not. It could be also loaded to the device later on. And this is, this is kind of interesting and sort of a game changer in the sense that, you know, architects, we develop drawings and plans in order to build buildings and structures, whereas with Fologram, it removes some of that in the sense that now you can basically give a contractor, a three dimensional model that they can view through a hololens and then begin to assemble simple things. So one of the examples that, that we’ve played around with is bricklaying so complex bricklaying. Brickstone always have to sort of stack neatly in a row, but you could begin to have other surface treatments based on the rotation of the brick within the wall. And so by, instead of drawing like a series of plans that begin to show that a Mason could, could load the model to the hololens, and then basically just place the brick within the framework that’s being projected in front of them on the construction site.

Jonathan Rule (14:49):
And so that can be done with bricks. It could be with another, a number of other things, pipe bending, wood bending, and so forth. And so there’s, there’s been a number of experiments at the college that, that have kind of looked into that. And I think that’s, it’s sort of an interesting way where these technologies, I think again, we have like the virtual but then also the, the augmented, I think might be of a more powerful connection when we talk about how things are actually built physically and how we’re leveraging these extended realities.

Jeremy Nelson (15:21):
Yeah, no, it’s, it’s, I think we’re, we’re very interested in exploring on where best do the technology solutions fit. Like, you know, how best do we deploy VR and what are the types of interactions? What are the type of learning goals? And then the other side from a practical standpoint or skill base, where does the augmented reality, the hololens work, come in and we’ve been having lots of discussions with folks across campus about that. And I just see so many opportunities with both, it’s pretty exciting. In terms of we’ve seen and, you know, best cases are exciting. Do you have any concerns or cautions as we move forward about the future of XR and teaching and learning?

Jonathan Rule (15:59):
Yeah, I mean, I think my, my biggest concern, and I think a lot of colleagues around the university share that as well is accessibility to this technology. And obviously there are certain that can be quite cost prohibitive. Again, the hololens is not cheap. VR headsets are becoming a little bit more affordable but still might be out of reach. And then also coupling that as not just the VR headset, but making sure you have a computer that could begin to run these things. So how do we make these things accessible to our students beyond labs? So as we begin to think about the development of, of these, these devices or development of software for these devices, how do, how do we make sure it has the broadest reach and doesn’t exclude parts of the student body or faculty as well. And it’s something that’s also I think that we have to think about as a university in general technology is constantly updating. So just making sure that we’re also staying ahead or with the curve, at least in the updating of the resources that we have for students at the college. I feel like you buy something today and it’s obsolete as soon as you purchase.

Jeremy Nelson (17:16):
Right, right. Yeah. We’ve seen that with some of the Oculus devices, the questis, can now be a refdesk with a USB cable. And yeah, I mean, we’re thinking about that broadly as well of this, you know, generating or procuring enough content and what types of devices that’ll run on. How do we secure enough for those devices? You know, we have a large visualization studio at the Duderstadt center. We have been in discussions with Taubman college about creating an XR lab there there’s other folks around campus that want to create, you know, these nodes of XR labs or facilities to access. And I think it’s an important question and it’s a, you know, it’s not simple. But I think that’s part of our mission as well, to help, help solve that.

Jonathan Rule (17:59):
Yeah, totally. And I mean, I think this will just like, beyond the sort of the hardware aspects of it, I think it’s also, thinking about like what this is being used for and sort of really kind of looking at what is being developed, how it’s being developed. And I think also like in my mindset they’re not necessarily a replacement for things. There’s certain ways that we do things and the virtual is, is again, it’s a simulation, but it doesn’t necessarily replace like reality. So I think just being cognisant of like how, how things are being used or what they’re being developed for and what the end goal of, of those are, I think is also important just to keep in mind.

Jeremy Nelson (18:41):
Yeah, for sure. It’s another technology. Well, what do you, what do you want to see Michigan do to enhance the education of students with XR technologies? What does the future look like to you and what would you like to see happen?

Jonathan Rule (18:53):
Sure. so I think, and, and maybe this is also just because I’m sort of new to this at the University of Michigan. And I believe the role of the center for Academic Innovation and what you’re bringing to the table right now is, is helping to generate more conversation across units on campus. And so I think just, just seeing that, like making sure that that is sustained and as we begin to develop, not only developing just for, within our units, but what are certain things that different faculty from, from different areas and different disciplines can begin to contribute to one another to begin to push the boundaries or understand this technology in new ways. I think sometimes like we, we kind of we’re very used to working on what we’re working on and understanding what our research is about.

Jonathan Rule (19:42):
And we kind of, we have these blinders on. And so if we can kind of take those off and begin to kind of look around us and say, Oh, well, maybe this person can begin to help me in this, or they understand it better than, than I do. How do we begin to kind of pull those, those together for enhancing these projects that we’re all working on? So I think that that’s sort of what I would like to see, how do you, how do you become a bigger catalyst for conversations and collaborations across campus?

Jeremy Nelson (20:10):
Yeah. I love it. That’s, that’s in line with my vision too, so that’s, that’s great. Yeah. Well, no, this is, this has been great. I love hearing more about your work and your history to get here. You know, who else should we be talking to? What other topics should we explore? You know, what’s of interest to you kind of in that vein of this cross-discipline approach?

Jonathan Rule (20:30):
Sure. so cross discipline that one’s, that one’s a tough one. I’d have to think about that for a little bit. I think, I think I still have my blinders on, I’m still thinking about it in the world of architecture.

Jeremy Nelson (20:42):
Or within Taubman or within your space.

Jonathan Rule (20:45):
Yeah, I think, I think Matias del Campo would be a good person to discuss this with, or Jose Sanchez who’s going to be coming on board in the fall. He would be a very interesting person to discuss it with. He also he’s developed Block’hood, which is, has been there for, for a while. A number of years now which is looking more at a, at an urban scale as opposed to the architectural scale. So I think that those two would be interesting people to discuss this with. And then I think also just reaching out maybe beyond the university and looking at what’s out there, sort of in the industry. So maybe speaking with architectural offices that are beginning to leverage this. So I know shop architects in New York, they’ve been working a lot with Unity on developing a Unity reflect, which is again an integration of augmented reality with BIM models, for construction visits BIM models being building information models. So maybe reaching out to them and seeing sort of like what their vision for this this is, or again the, the guys at Fologram, might be a really interesting discussion as well, which is again, using more of the augmented reality aspects of extended realities to develop architectural space.

Jeremy Nelson (22:07):
Yeah, those are great. They’re on my list. Well, this has been fascinating. I loved the conversation. I look forward to the, our work moving forward and just thank you for your time and really appreciate it.

Jonathan Rule (22:17):
No, I do too. Thank you. Thank you so much, Jeremy. And yeah, looking forward to seeing how this all develops.

Jeremy Nelson (22:28):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.