Jeremy Nelson, Director of XR Initiative 

In this week’s MiXR Studios podcast, we talk with Matias del Campo, associate professor of architecture and urban planning, about his journey into XR and AR for construction architecture. Matias teaches courses in Generative Design Computing and Virtual Engagement at the University of Michigan’s Taubman College of Architecture and Urban Planning.

We talk about Matias’s early work with the Hololens and how he began working with the company Fologram to teach. We explore his work history dating back to the 1990s and how the advances in electrical engineering and computer science have dramatically accelerated the technologies that can enhance architecture workflows. Many architecture students learn to create 3D models of their designs in a software application called Rhinoceros, but the process to visualize those models on a Microsoft HoloLens is not easy. In partnering with Fologram, the opportunities for students to build and visualize can be done in a matter of hours versus days.

Students in Matias’s class have been able to get hands-on experience with the HoloLens to assemble their designs in the real world with a level of precision that is substantially higher than traditional methods. The students have reacted favorably to this style of teaching and learning and have been able to think on a larger scale and iterate much quicker. We talk about a specific project where students take raw materials and are able to bend them into a design in less than 30 minutes that would have in the past taken multiple days.

We explore how this technology is being used by industry to change how construction is happening today and how the tools and techniques that students are learning will prepare them to shape the future. Matias is very interested in interdisciplinary work across the university and shares some of the work he is doing with colleagues at Michigan Robotics. There is a natural synergy between work in artificial intelligence and machine learning that influences design and architecture and Matias is at the forefront of that. He has helped designed a robot garden at the new Michigan robotics building for bipedal robots that is a “robotic garden designed by robots for robots.”

I really enjoyed talking with Matias and learning about his work and how he is at the bleeding edge of architectural design and teaching the next generation of architects to shape the future. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 10

Jeremy Nelson (00:05):
Hello, I’m Jeremy Nelson. And today we are talking with Matias del Campo, who is an associate professor of architecture at the University of Michigan’s A Alfred Taubman College of Architecture and Urban Planning. We are talking about his work with augmented reality and the HoloLens to enhance construction design and dramatically improve precision in his students’ work and efficacy. Coming up next in our mixr podcast.

Jeremy Nelson (00:38):
Well, hello Matias. Thank you for joining us today. We’re looking forward to hearing from you.

Matias del Campo (00:43):
Hi Jeremy. Thanks a lot for the invitation to join your podcast.

Jeremy Nelson (00:46):
Yeah, no, it’s super exciting. I remember seeing some of your work early on when I started and got to hear a little bit more about your work and it was super exciting and, and we haven’t chatted in a little bit, so would love for you to share your journey into XR. What you’ve been doing recently, what do you have coming up? Folks would find it fascinating.

Matias del Campo (01:08):
Absolutely. Happy to share some of the things we’ve been doing around the Taubman college. So maybe I should start a little bit to how this whole thing started for us for this journey, especially at Taubman college. So I personally have been involved with augmented reality and especially virtual reality pretty much since the nineties in, in more or less extent. So it was somehow always following along the work that I’ve been doing, but it has absolutely taken it took off absolutely in the last two years through to the progress in terms of technology. So I became very interested in using augmented reality in architecture for a variety of different reasons. And what came in extremely, well was extremely helpful in that extent was the involvement of computer science and electrical engineering at the university because of their ability to have access to the necessary hardware. But also of course the development of tool sets for them. And so they were very generous in borrowing my studio and my seminars several HoloLenses, so that we had access to hardware. And at the same time, we were supported by a company that is specialized in creating augmented reality applications for, specifically for architecture.

Jeremy Nelson (02:30):
How did you find them, or how did you, you know, had you met them back in the nineties or how did you get involved with that?

Matias del Campo (02:36):
Not, not that far back, but rather recently. I mean, they, they are graduates from RMIT, Melbourne. And they had been tampering around with augmented reality through, throughout their studying time and basically went pro afterwards developing a software that allows to use the HoloLens seamlessly with Rhino, which is one of the main tools in terms of workflows in architecture these days.

Jeremy Nelson (03:06):
It sounds perfect.

Matias del Campo (03:06):
So he facilitated the process very, very much. Yeah. But then I basically met them in Melbourne. I had been teaching there occasionally and we stayed in touch. And funny, it’s a funny story actually, because I was desperately preparing a workshop with the HoloLens and there are processes to connect the HoloLens to 3D models, but it’s rather cumbersome. And I learned one of those processes and was happy I can do this workshop. And I think about a week before the workshop, the whole workflow was changed by the company that is doing the software and it completely destroyed my workshop.

Matias del Campo (03:43):
Yeah. And I was really desperate. So I just texted these guys in Melbourne and told them, look, guys, I have this problem. You know how to get around that to make it run? And they’re like, Hmm, did you actually already try our software? And I was like, you think, you do the software? Oh yeah, we have something. Well, happy to try it. So they send it over and literally five minutes later, everything was working fine.

Jeremy Nelson (04:08):
That’s amazing. So what’s the name of the software? What does it do exactly?

Matias del Campo (04:12):
It’s called Fologram. And basically what it does it can connect the HoloLens directly with Rhino and with a model that you have in Rhino. And it shows you that modeling one-to-one scale and there’s a variety of different tools they have developed around it in order to aid in fabrication. And I was at the beginning a little bit skeptic because of the precision necessary in architecture to do things.

Matias del Campo (04:36):
But I was surprised how well it actually works and how precise you can go with it. So this was really an extremely helpful tool for me. And it has come quite a long way since I first used it. And students love it because it’s just so easy to use and they really can interface with their hardware very, very quickly. Yeah. So they can learn it quickly, which of course facilitates what you can do with it instead of just learning the technology behind it.

Jeremy Nelson (05:03):
That’s fascinating. So how did, how, is it more research are you using it for teaching? How do you engage with the tool or how do the students, you know, what are they hoping to learn in your course?

Matias del Campo (05:13):
So what I’m focusing in in the course is two specific areas like on the one side using the HoloLens or augmented reality as an aid of construction so that you can train people for assembly.

Matias del Campo (05:26):
For example, if you have to assemble complex pieces with each other, it normally it used to, you need a bunch of plans or a lot of documentation actually to make it correct on site. But the HoloLens really facilitates the process of teaching a construction crew how to set things up and how to build them. That’s one thing that helps a lot, especially when it comes to more articulated and complex objects in architectures. There’s, there’s always this this let’s say angst that you could do something wrong because it’s complex pieces that come together. But using, using a hologram shows you exactly where in space, which piece goes. And that of course facilitates a lot of the process. That’s one side and the other side is basically understanding it it’s a cultural tool. Like, you know, you’re basically creating augmented realities around you in the existing space and between these two poles, the courses are oscillating.

Jeremy Nelson (06:23):
Well, so yeah. Tell me more about that. So how, how does that work? How are the students, you know, are they, they’re using the HoloLens. Are they looking at the structure? Are they looking at the materials and then they’re putting them together? Or is it just a visualize it and then they go do it?

Matias del Campo (06:36):
These things all go hand in hand. So there’s a couple of videos out there from my courses that shows this really very clearly where for example, students create a digital model in 3D and then have to figure out which are the components that are necessary to build this structure. And then they use, for example, in our fab lab, the robots to fabricate the pieces. Yeah? And then use augmented reality to solder them together because they exactly see where in space they go and they actually just mount them together loosely so that they fit with the 3D model in space and then weld them together. And the results are incredibly promising and the precision is very high. It’s very easy for them to do it. I had another student, for example, who, who was folding components manually, and you could do this before using paper templates and to use, we’re constantly, you know, comparing the paper with your model. But now, now, instead of using a piece of paper and the print, you literally can see through the HoloLens in 3D where this piece is in space and you can manipulate with your hand. So it fits that form. It’s, it’s really incredible what you can do with the things.

Jeremy Nelson (07:49):
Ah, that’s fascinating. So, so did you teach this course before without augmented reality or has it always been augmented reality from the beginning?

Matias del Campo (07:57):
We had other tools. I mean, we, let me put it this way. I think the development of using augmented reality really took off in the last couple of years where you had, you know, tools like Unity or these softwares that allow you to do augmented reality applications using, for example, your mobile phone. So we, we actually did that. We created some apps that students can use that are triggered, for example, from, with images that are around you and the chose shapes in space. So there was always this idea of understanding that you can use augmented reality as, as an, as an amplification of space and architectural experience. So there’s like this overlay of information possible in space that is only accessible through augmented reality and normally it’s invisible. So it’s really beautiful that you have like a variety of different ways to create a narrative in architecture now through these kind of tool sets. So we’ve been doing this for a couple of years, but the HoloLens really changed a lot in terms of, you know, transforming that notion into something that can help to create a physical environment through the use of it. Yeah. And I like this combination now between the two, like for example, creating something, using the HoloLens in terms of application, but populating that structure again with things that you only see in the augmented reality. So you can combine these ideas.

Jeremy Nelson (09:19):
Oh, that’s super cool. So what are the, what do the students enjoy the most or what have been some interesting patterns or learnings you’ve witnessed?

Matias del Campo (09:28):
I think they like it a lot when they can build something very big, very fast, that that brings a lot of gratification where normally you would need to measure and measure again and make sure that things are compliant to the planning that you made. That suddenly because of the use of the HoloLens plans and sections and detailed things are obsolete. You don’t have to have this documentation of 50 pages to do a small installation anymore. Yeah? Suddenly all really works in 3D in space. In real time, you can, can have access to numbers on, for example, if you see that a structure in front of you and every single piece has a number in a, in the letter defining which piece it is in space, and you, you look next to you and on the ground, you have the physical piece with a little sticker with the number and the letter, just pick it up, put it there and you’re done instead of having to go through 50 pages of planning to find where is that piece and where does it go?

Jeremy Nelson (10:26):
That’s great. So, yeah, you’re basically just almost snapping it together at that point and putting it in order. What types of work have you seen students do that kind of blew your mind or was just completely fascinating that you didn’t think was possible?

Matias del Campo (10:41):
One thing which I thought was absolutely great was when they created a dome out of pipes, of pipes that you can bend. Yeah. And so there’s like these long plastic pipes and they were scattered on the floor, yeah? And it looked like nothing and a couple of cables and some little pieces of laser cutter acrylic. And then about 20 minutes later, there was this beautiful, intricate very woven structure standing in front of us, which we literally could inhabit because we can walk into it.

Jeremy Nelson (11:13):
Wow. 20 minutes?

Matias del Campo (11:15):
It happened in no time. It was really that because the tubes were so easy to bend and they only had to look on the augmented reality model to see how much they have to bend it.

Jeremy Nelson (11:25):
Wow.

Matias del Campo (11:26):
Yes. Granted, there were, there were a lot of simulations beforehand about the, you know, the, how much does the plastic bend? How does it look when you bend it? So they tested a lot of things, but the final assembly was mind blowing and a beautiful structure, very elegant.

Jeremy Nelson (11:42):
How long do you think that would have taken someone if they didn’t have AR they had to do it just on the paper? I mean, any estimates?

Matias del Campo (11:50):
Uh, half a day, for sure.

Jeremy Nelson (11:52):
Wow. Wow.

Matias del Campo (11:53):
And also probably wouldn’t have been that precise. Yeah. More bending here, less bending there. The curve is not perfect and so on. So all of the things that really create this wow effect that it’s such a beautiful, pristine structure, it’s, that’s really hard to achieve. And if you want to make it that precise, they would have probably have taken two days or so to get it there.

Jeremy Nelson (12:13):
Wow. That’s amazing. Two days down to 20 minutes, that’s pretty fascinating. Are you starting to see this out in industry now? The people using the technology or beginning to do work in AR?

Matias del Campo (12:25):
I have to say that, of course, because of the current situation, it has become a little bit more complicated to really use as a group augmented reality. But on the other hand where I see it emerging now as a tool is when it is about design meetings. When several people have to look at the same design object and they can do that distributed throughout the planet using augmented reality. So they don’t have to physically meet anymore to discuss a physical object.

Jeremy Nelson (12:54):
Right. Yeah, no, that’s, yeah. We’ve been exploring a couple projects of how do you look at shared 3D models from multiple devices. I talked to some folks at, in the medical space at Case Western, and they’re doing an anatomy course in the HoloLens and they were able to go remote very quickly and continue their courses. Cause the students could pick up, you know, begin class again all wherever they were through the HoloLens. That’s pretty fascinating.

Matias del Campo (13:19):
Yeah. And I’ve seen actually also applying it to more traditional craftsmanship. So, which I think is fascinating, for example bricklaying. Yeah. It’s such a mundane process, right? But because of the use of HoloLens, they can start to create far more intricate patterns with bricks manually that was possible till now of course it was possible, but they would have to, it would have taken much more time to get it right and precise. So we know all these examples of robots laying bricks that have these beautiful patterns and undulating and so on. But I like the idea that you can go back to a craftsman who basically just puts bricks together. And because of the use of the HoloLens, you can keep his business and his craftsmanship. A new dimension. Yeah. And the new possibility, a new lease on life. I think that’s great.

Jeremy Nelson (14:11):
Yeah. No, that’s, that’s, you’re really augmenting the craft then. Well, no, those are super exciting examples. I mean the work sounds fascinating. I mean, do you have any concerns about the technology or the future of XR for teaching and learning?

Matias del Campo (14:27):
No, I think it’s a, it’s a great opportunity. Of course, there’s always somebody who will abuse it. That’s like with any invention in our history as humans, but all, in all, I think that the, the benefits outweigh the, the problems. Yeah. And that, especially where, when I’m saying that you can combine traditional craftsmanship and, and advanced technology, I think that’s a very, very interesting combination because it’s fair to both areas. So it’s not only about progressing forward, but actually to take along the people that are with you on the journey. And there’s still a lot to do. I mean, you have to think about that in terms of augmented reality, what we are using right now and not even products yet, it’s still in beta. So I’m looking forward to the point where it becomes really a widespread phenomenon and goes really beyond just a couple of people trying to understand what we can do with it.

Jeremy Nelson (15:19):
Well, I know you you’ve been at the forefront in the AR space here at Michigan. What do you want to see us do more broadly to enhance the education of students with XR?

Matias del Campo (15:28):
I think what would be absolutely great is more interdisciplinary work with people who are using those technologies because this kind of cross pollination is what creates new ideas, what you can actually do with this technology. So I would rather like to see the possibility to create mixers between different departments that are working with the same technology.

Jeremy Nelson (15:50):
And so what’s up next for you? What type of research or work are you doing and is it evolving at all because of the current COVID situation?

Matias del Campo (15:59):
What I’ve been focusing on now for the last two years actually is, is using artificial intelligence in architectural design, machine learning. Yeah. And the funny thing is that that’s also a result of a collaboration with Michigan robotics and computer science. And what I learned there about, for example, machine vision is something that can be implemented also with XR. Yeah? So the, the way how machines understand our environment and how it could inform us as humans through the use of XR is certainly a very exciting field of exploration.

Jeremy Nelson (16:34):
Yeah. That sounds fascinating. Have you been able to incorporate that into any of your work yet? Or is it more theoretical at this point?

Matias del Campo (16:42):
The, no, the, the, the AI work has yielded results that are tangible. So we are currently building the so-called robot garden for Michigan robotics and the new building, which is basically a test ground for their bipedal robots. So it’s, it’s a genuinely a post-human architecture. It’s not for humans. It’s for robots.

Jeremy Nelson (17:04):
Yeah, yeah. Right.

Matias del Campo (17:05):
So, and we used, actually the techniques were developed in artificial intelligence, which are several neural network techniques to design that garden. So I, I like to joke around with with Jessy Grizzle, the director of Michigan robotics, that the robotic garden was designed by robots for robots.

Jeremy Nelson (17:24):
Yeah. That’s great. When is that supposed to be open? I know that was supposed to be open soon, right? When the gardens targeted for?

Matias del Campo (17:32):
Yeah. The, it was supposed to be opened in May. Of course, that did not happen, but we are, I guess it’s going to be open in fall. I hope at least to see that running then we will see.

Jeremy Nelson (17:43):
Okay. Yeah. We’ll have to go check that out. That sounds pretty fascinating. This has been great. I think I really enjoyed talking with you learning more about your work, you know, are there other areas or other topics that you think we should be exploring in this podcast? Other folks we should be talking to?

Matias del Campo (18:01):
Yes. Actually there is several other colleagues in Taubman college who are all, we’ve basically created this sort of XR group at Taubman because there are several courses that are interested in teaching that, and that goes from really virtual reality to augmented reality. So different flavors. Yeah. And Tom Moran would be certainly somebody to talk about in our school about that. So I can give you certainly more names too.

Jeremy Nelson (18:25):
Yeah, yeah, no, I’d love to, you know, we’ve talked with Jonathan Rule, we’re working on an XR innovation project with him called augmented tectonics. So yeah. I’d love to talk with more folks at Taubman. And I think there’s lots of opportunities in, in architecture for this technology.

Matias del Campo (18:40):
Exactly. And we’ve taking that very consciously. So there is support from our Dean for this. He, he’s trying to understand that there’s a whole ecology of people interested in these topics in the college and that it can become a strength also of our school.

Jeremy Nelson (18:55):
For sure. Definitely. Well, whatever we can do to help support that. So yeah. No. Well, thank you so much for spending time with us sharing your work. It’s it’s super fascinating. I can’t wait to see it when we get back to campus and I appreciate it.

Matias del Campo (19:11):
Yeah. Thanks for the invitation was a pleasure to be here and talk to you, Jeremy.

Jeremy Nelson (19:15):
All right. Have a great day.

Matias del Campo (19:16):
You too. Bye Jeremy.

Jeremy Nelson (19:23):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ai.umich.edu/xr.