Jeremy Nelson, Director of XR Initiative

In this week’s MiXR Studios podcast, we talk with Simone Sessolo, a lecturer IV at the Sweetland Center for Writing and the Digital Studies Institute at the University of Michigan’s College of Literature Science, and the Arts. Simone discusses his work in teaching composition and narrative writing with augmented reality. His work focuses around the interactions that exist between computers and traditional media. In talking with Lisa Nakamura from the Digital Studies Institute, he was intrigued by the university’s investments in augmented and virtual reality. This led him to create a 200-level writing course at U-M called Creating Narratives in Augmented Reality. We explore how he thinks about storytelling and digital technologies and what he would like to see the future look like at the university.

In May 2020, Simone was awarded the Inaugural Digital Studies Institute Teaching Award. He teaches courses that range from Introduction to College Writing to Dissertation Writing, so across the spectrum from freshmen to graduate students. Additionally, he chairs the Digital and Social Media Curriculum Committee, and teaches some of the Writing 200 and Writing 201 courses where students analyze and apply rhetorical principles in their writing with digital media. These courses ask students to use multimodal forms of communication and become more informed and critical consumers of whatever is shared through digital channels.

Like other faculty here at U-M, Simone was intrigued and captivated by the energy around Pokemon GO and the level of engagement people had. He began to explore how he could bring this type of technology to his courses and how students could transform storytelling with augmented reality. The course he designed was built around three main modules: Theory, Composition, and User Experience. Simone shares how he takes students through lessons to understand plot, narrative time and paratext, and how this leads into an example of taking a short story and turning it into an infographic. Students are then asked to deconstruct the infographic and think about how they can place that information in three dimensional space.



Metaverse is an AR authoring application that can be used to create AR experiences

Simone isn’t a software developer, and he wanted to find an application that let students create AR experiences with little or no coding requirements. He was inspired by platforms like WordPress that allow people to create very complex digital platforms without the need for complex coding. This led him to the Metaverse application that is focused around creating stories in augmented reality without having to code. This application is quite robust and has a great set of resources to help teach people how to create applications and incorporate interactive elements like tours, events, quizzes, games, and, of course, stories.

This first course, Creating Narratives in Augmented Reality, launched in Fall 2019 and was centered around students working in groups to create an AR story for a club or organization at the university. The students approached groups across campus and coalesced around four areas of focus.

• North Quad Dining
• The Summer Bridge Scholars Program (Angell Hall)
• The Peony Garden at Nichols Arboretum
• The Computer Showcase (located in the Shapiro Library)

These group projects highlighted the diversity of perspectives that each student brought to the course and the project. Each group had to interview people at the organization, understand the problem they were trying to solve or goals they were trying to achieve, and design and build the application. Each of these projects was well received by the organizations on campus and the apps were on track to be used, until COVID hit.

Simone is teaching the course again this semester, Fall 2020, and everyone has had to adapt to the current situation of being remote and working more independently. His direction this time was to have students create an experience at home and consider the types of things that everyone has in their home. The goal is to take something that is in your home and build a story around it that is personal to help, in some small way, make this experience more pleasant for everyone. As with the course last time, students will be asked to review each other’s work and provide feedback on the user experience. This will create a social aspect to students sharing their work and stories about their home with others.

We conclude our conversation by discussing some of the challenges and concerns with digital technologies like XR and how even basic technologies like laptop computers and smartphones are not always a given for every student. At some point in the future, XR technologies will reach a price point or be integrated into smartphones to allow for more ubiquity, but today, Simone is advocating for a basic computing device like a Chromebook, if the Student’s don’t have one already. We discussed how the XR Initiative is making Oculus Quests available for students to check out for courses starting in the Winter 2021 semester and how that can start to help move the needle forward.

I very much enjoyed my conversation with Simone and learned so much about how you can take a “traditional” writing requirement and turn it into an interactive digital experience. Please share with us what you would like to learn more about in the XR space at [email protected].

Subscribe on Apple Podcast | Spotify

Transcript: MiXR Studios, Episode 26

Jeremy Nelson (00:10):
Hello, I’m Jeremy Nelson. Today. We are talking with Simone Sessolo, who is a lecturer IV, at the Sweetland Center for Writing at the Digital Studies Institute at the University of Michigan’s College of Literature Science and the Arts. We are talking about his work and teaching narrative writing and using augmented reality as a medium to explore composition and narrative coming up next in our MiXR podcast.

Jeremy Nelson (00:52):
Good afternoon, Simone Welcome.

Simone Sessolo (00:54):
Thank you. Pleasure to be here.

Jeremy Nelson (00:57):
Yeah, thanks for, for making the time and being a guest on the podcast. I was excited when we met recently and learned about your, your work here at the university in your courses, and yeah, love, you know, why don’t you start with how you got into this, what courses you teach, just kind of what brought you to using XR as a medium?

Simone Sessolo (01:19):
Sure. Yeah. Um, well, first of all, I, I must admit that, um, my background is not in XR or digital communication or information. Um, my background is in narrative and writing. My PhD is in comparative literature with, um, a focus on narratology and classical rhetoric. So it might seem that I do something completely different from XR, but I’ve always been interested in, in the possible interactions that exist between computers and traditional media. Um, considering if there are similarities between the writing process that writers follow when they put pen on paper or when they type on a keyboard, for example. Uh, so, uh, it’s, it’s those moments of thinking composing and revising that writers go through, um, as they engage in, in communication acts that that are interesting to me. And, and I got started in, in augmented reality, um, because during one meeting in the Digital Studies Institute, our director Lisa Nakamura, uh, was talking about the university’s interest in virtual and augmented reality.

Simone Sessolo (02:38):
And so I thought, wouldn’t it be cool to think about a writing class that incorporates some of these technologies and at that time? Yeah. Yeah. And, and at that time I was taking the bus to, to come to campus when we could come to campus. And even though I’ve never been a gamer, um, I was, I was captured by seeing people playing Pokemon GO on their phones. And I thought about the people behind that, uh, the writers and of that game. And so I thought, well, students might find it fun and exciting, um, to write something similar. So I start looking, looking around, I discovered things that I could use in my class and, and that’s how it started.

Jeremy Nelson (03:31):
Ah, that’s really cool. So was it a, was it a course you were already teaching, was this a new course you were proposing? How did that play out?

Simone Sessolo (03:39):
It was, um, both partly so, um, at Sweetland and in digital studies, I always teach some of our writing and digital 200 and 201 courses, um, where students analyze and apply rhetorical principles in writing with digital media. Um, and I was teaching some of the classes that I taught previously were about the selfie and about memes, but that discussion in the digital studies Institute, uh, prompted me to create this, this new class that, um, uh, I’m teaching this semester and it’s titled creating narratives in augmented reality.

Jeremy Nelson (04:29):
Oh, that’s cool. Yeah. It’s kind of like the next evolution of that storytelling and I think a lot of people go ahead.

Simone Sessolo (04:37):
Absolutely. Yeah. And, and like the reason behind me designing this class was because I realized that augmented realities is increasingly becoming a presence in, in our everyday lives. Uh, it’s not just games. We play on our phones, but it’s sites. We visit to buy items from, from glasses to shoes to, to going to stores that sell furniture. And then we can figure out how the furniture look in, in our home. So, uh, we all experienced aspects of augmented reality, uh, at one point or another in our lives, but, and here’s the problem. This experience risks being a passive one with users being devoid of the literacy to interpret what the narrative or the message behind these sites is. So, yeah. So in creating narratives in augmented reality, um, I want to focus on how a narrative in designed in augmented reality participates in, in rhetorical and social discourses. And, um, I want to see those experiences as, as rhetorical acts and my students engage with issues of authorships authorship and an audience in those narrative narratives.

Jeremy Nelson (06:09):
Hmm. So, uh, that’s really cool. So in your mind, did you have something in mind when you, when you thought about this or was there an example of how you thought it might go and then we’ll get into, you know, what the students are actually doing this point? Um, how did that, what did that look like in your mind before it started? Yeah.

Simone Sessolo (06:27):
Um, I knew I wanted to include some theory, some composition and some user experience. So, uh, so I started from these three modules and, and in the first one, um, we mostly focused on traditional narrative concepts, such as plot, narrative time, uh, paratext the narrator and so on and so forth. And we’ll look at how these concepts work in texts like short stories and novels this semester, we’re reading some of James Joyce’s Dubliners, for example. Uh, but we also look at, uh, movies and infographics and we try to see if we can notice similar things happening in augmented reality narratives as well, uh, for the composition module. Um, I asked my students to create an augmented reality experience using the app metaverse, uh, which is a platform that allows to design these experiences without having to know how to code. So, um, it, it similar to what Wix or WordPress do for websites.

Simone Sessolo (07:44):
Um, in, in the first time I taught this class, I had students work in groups and each group, uh, decided to get in touch with a club or organization at U of M and create an augmented reality experience for those organizations. Um, that, that semester, my students, um, worked with, uh, North Quad dining, um, Angell hall, particularly the bridge program, the Peoinies garden and the computer showcase store in the, in the UGLI, um, this semester because of, um, the pandemic, I’m asking my students to create a home experience. So, um, considering types of things that people have in their homes, and like everybody has a bedroom. Everybody has, um, a kitchen or kitchenette or somewhere, or, or, uh, like a microwave. Everybody has books. And, um, considering those items as several points of information that can be augmented by what the creator of the augmented reality experience, uh, decides. So the goal is to make the sheltering in place, experience more pleasant. And, you know, the the ideal behind it is to contribute in a small way, uh, to bettering someone else’s everyday life. So there’s this social aspect as well.

Jeremy Nelson (09:32):
That’s, that’s great. Yeah. I like that, uh, XR for good.

Simone Sessolo (09:36):
Exactly.

Jeremy Nelson (09:38):
So, I. Yeah. How are, how are students reacting to it? What was the interest in the course? How large is the course?

Simone Sessolo (09:46):
The course is capped at 20 this semester. I have 18 students. And, um, and as I said, this is the second time I’m teaching this class. And so far I’ve received very positive responses. Uh, what I, what I like particularly is the, uh, the disciplinary diversity of my students and the students who take this class. Um, some of my students are majoring in English or the humanities. Some are minoring in writing and digital studies, and there are also, uh, some STEM students. So that is absolutely wonderful because students get so many different perspectives when they collaborate. And, and each one contributes to augmenting each other’s knowledge. This is actually, uh, what happens to the third module in the class, which is mostly about user experience. So students test each other’s AR experiences, and then they give each other feedback. And, um, because they come from such different, uh, academic backgrounds, um, they receive very, very diverse suggestions. And that’s wonderful.

Jeremy Nelson (11:04):
Yeah. I love that. I think that’s, that’s at the root of a lot of what I’m seeing, how these experiences are built and the team that needs to come together of kind of the, the narration or the storytelling. You have, the deep technical, you know, programming, you have the art, you have audio, the production, like it’s a, it’s a whole team. It’s, it’s great to see that naturally, maybe you’re playing out in your course.

Simone Sessolo (11:28):
Yeah. And, and, and again, um, I want to, to stress that I am not an expert in coding or in, in working on the technical aspect of, uh, augmented reality. So, um, the reason I like the app that we use in my class metaverse is that it allows anybody as long as they own a computer and a smartphone to create these experiences. And what’s, what’s very, um, interesting to me and positive is that, uh, the creative process for these experiences is based on a linear composition process, that most people are already familiar with, like telling a story. Um, so, so in a way that the project reminds me of, of creating a comic book, because users go to the metaverse website and they start with creating a storyboard that initially is made up of sequential scenes. So, uh, users can, can really have a sense of where and how they want their story to progress.

Simone Sessolo (12:55):
And at the same time, users can also create shortcuts and link scenes, non sequentially, and, um, like what happens in hyper texts, for example, so they can rely on, on their previous knowledge of navigating websites, uh, to, to see in their mind how this experience might, uh, might be realized in, in, in real life. Uh, but I think that having the possibility to access the storyboard and to see that Tableau of the whole experience allows them to attach, um, something that they might at first consider arcane to something that is more familiar to them, a text experience, and that is inclusive and inviting at the same time.

Jeremy Nelson (13:52):
Sure, sure. Yeah. I mean, it’s important. We, we’re building a number of, you know, VR and AR experiences for courses that we funded last fall and, you know, we’re, we’re storyboarding them. We’ve, we’ve hired some student designers to help us think through that. And because when you’re designing in 3d space, it’s, it’s similar, but a bit different where you can go and what you can interact with. And, um, that’s off opens other opportunities or where do you place texts or how do you interact? Like the, yeah, yeah. The UI is a little bit different and, or not as well understood as is web and other formats.

Simone Sessolo (14:26):
Yeah, yeah. About that. We have, um, a series of, of exercises in my classes that it’s, it’s always, of course easier to do in person, but, um, we start from reading a chapter of a book it’s the Cambridge introduction to narrative. Um, and it’s the chapter on the narrator. And then in class, I asked students to create an infographic, um, that reports the information that they have learned in the chapter. So in that there is the transition from only text to, uh, text and images. And then, uh, the next step is, okay, you have your infographic, let’s cut it out. The parts of information. And we are in this room, let’s imagine that this room is a museum about the narrator. Where would you put that piece of text in the room and why is it located there? And that creates the 3D experience to, okay, this is if I have an AR experience, that’s where I would add the code with the information that wants to augment that specific, that specific place.

Jeremy Nelson (15:43):
Yeah. I like that that’s scaffolding or transitioning into that way of thinking. It’s great. What were some of the projects that surprised you the most, or you were most proud of in that first, last semester? First one,

Simone Sessolo (15:57):
All four, all four were good. And I’m not saying this to be correct or anything. I really think that, uh, they were wonderful projects. I think that group projects always allow the best in everyone to come out and, um, and, and collaboration becomes organic. So, um, they were very different projects. Um, the, the one about the peonies garden for example, was outside. So you had to go through the garden and you had this, this walking experience. Um, the, the Angell Hall experience was great because it was particularly, um, directed to students in Bridge. And so, uh, students who are not familiar at all with U of M, and so they, they, the experience gave them some information of about where to go, who to contact, contact, where to find staff. Um, the computer showcase one was the, the most commercial in a way, because it had a lot of information about, um, of course laptops and tablets. And it was very interactive in, um, going from the experience itself to websites, to creating choices. Um, and the great thing about these experience, these experiences is that they are interactive. There are games in the experience, um, like to reach the next piece of information, you have to answer a couple of questions or find something. So it’s, it’s, um, it’s very gameful, if I can, if I can say that. Sure, sure. Yeah.

Jeremy Nelson (17:56):
That’s cool. So did they have, did they have to work with staff at computer showcase or in the dining hall, or like, how did they, was it like anchored, like, did they use spatial anchoring to like, how do I know, how does the phone know that this is the laptop I’m supposed to drop this? Or what, how did that, how did that play out from a technical standpoint?

Simone Sessolo (18:17):
Well, uh, at first they had to go contact that organization, uh, set up time for an interview. And, and of course, since this experience was for that organization, they had to inquire what that organization wanted them to show. It wasn’t just their work. They were working for, uh, either North Quad dining or, or computer showcase. So, um, yeah, they, they, they have to do some research, uh, in terms of the history of that organization, uh, the audience for that organization, what people going to those places, uh, usually expect and, and what would make that experience good for that particular audience? Yeah, yeah. This semester, we’ll see how it goes, because again, it’s this semester they’re, they’re working, uh, individually and so sure, sure. Yeah. But I’m, I’m confident I am, um, I’m looking forward to see what they write and what they create and to test them out. Yeah,

Jeremy Nelson (19:27):
Yeah, no, I mean, I think about, you know, these technologies, you know, there another tool to experience the world or experience story, or, you know, learning in a new way. And so it’s just kind of that you still need the root of, you know, understanding your audience, understanding the story, the narrative, like those are still a key. Yes. It’s, you know, making these enjoyable, successful. Um, interesting.

Simone Sessolo (19:53):
Yeah. And, and, you know, um, whenever we deal with, with new technologies, particularly in the classroom, um, some students, um, learn better with them. Some students, students might, uh, find, uh, find it more difficult. So, so there are risks in always risks and using new technologies in the classroom. But, um, but there are risks in everything. And, and, and I think that, um, being open with students and saying, you know, folks, some glitches are about to happen and not everything will go smoothly. And when, when something happens, we’ll figure it out and we’ll fix it together. And if we can, if we can’t, we will have learned something along the way, no matter what. So, um, so I, I like to see these writing and digital 200 classrooms that I offer, not as a place where knowledge is transmitted from, from teacher to pupil, but as a place where knowledge is, is always, uh, imminently, we might say collaboratively created. So, so the risks are, are part of the learning process.

Jeremy Nelson (21:18):
I love that. That’s great. That’s, uh, it’s really engaging. I would imagine for students and an inviting and in a different way, I mean, you started to mention some concerns. Did you run into any students that either didn’t have a smartphone or the technology didn’t work on their device or, um, you know, any equity issues?

Simone Sessolo (21:38):
Yes. It happens. Yes. Yeah. It happens. Um, not, you know, not having a smartphone or, or a laptop, uh, but maybe the memory in their phone is full and, you know, they, they have to, um, find ways around it. So, um, if there’s one thing that I would like to see happening at the university is, um, investing more in a widespread distribution of hardware, and I’m not talking about, you know, all Oculus or anything that expensive, but, you know, maybe each student receives a Chromebook when, when they enroll. Um, I don’t think that would be prohibitively expensive. Um, but it would set the basis for, for accessibility, uh, to acquire a stronger digital literacy. So, yeah,

Jeremy Nelson (22:44):
Yeah, yeah. They, they just did that with my children. Like every student got a Chromebook or an iPad at Ann Arbor public schools, they passed out 18,000 of them. Yeah. Pretty substantial. Yeah. Well, to your point, we, we are trying to make more, you know, part of my role in the XR Initiative is, you know, and understanding where the needs are. And a lot of the first round is like, you know, creating content or creating capacity for content. So it’s, it’s great to talk with, with folks like yourselves that are already doing that through your teaching. You’re, you’re creating content, creators and innovators in this space. We are in the process of making, uh, 25 Quests available for checkout courses through the Shapiro library. So mostly just to see, you know, how do students engage with them? How does that around the projects we’re building, then they’ll be able to run on the Quest.

Jeremy Nelson (23:33):
So, you know, how’s that going to work? What’s the cleaning protocols, you know, we’re going through very rigorous cleaning and making sure they’re safe and, you know, check them out, check them back in, what are the challenges we have? You know, it’s, I don’t, we’re not at a point yet where, you know, it’s a device somebody would need to purchase for that course. Right. There’s not enough content that makes that absolutely a necessity necessity. I don’t know if it ever will. Right. But yeah, the advantage of the, the D with the visualization studio on North campus that had 26 workstations, you know, they’re opening back up with, with limited capacity because of COVID, but, you know, hopefully that’ll be back to full strength, but we wanted to offer more options for students to continue to explore with this technology.

Simone Sessolo (24:19):
Yeah. And, you know, um, maybe in the future, uh, you know, writing for example is a required course, everybody at Michigan needs to take a first year writing requirement and then upper level writing requirement, which, you know, for us, at Sweetland is wonderful because we have, um, we can work with a lot of students. And I wonder if, you know, maybe, um, coding becoming part of, um, the requirements that students have to take to graduate might, um, Hm. Might even strengthen this, this digital literacy that is so necessary.

Jeremy Nelson (25:01):
Yeah. I’m as a computer engineer. I, I understand that. I, you know, it’s, it’s hard to find a part of civilization now. That’s not touched by software, right? Yeah. It seems to have permeated everything, um, and knowing how to take your area of specialty plus, you know, software and blending those together and manipulating, it would be powerful, um, in terms of creating the future. Well, where do you want to see, you know, kind of in your work or in this technology, what would you like it to, where would you like to see it go? Or what do you want to explore next?

Simone Sessolo (25:36):
Um, I don’t know. I want to see how this class goes in the conditions we are in. So, um, I’m teaching completely remotely, uh, this semester. So I want to see how that turns out and then, um, you know, I’ll, I’ll think about something else. Um, I’m very much a day by day, uh, kind of person.

Jeremy Nelson (26:06):
Sure. Well, yeah, a lot, a lot of very variables right now with this current situation, for sure. Yeah. Um, well, this is, this has been a great conversation. I really appreciate you spending time with us, you know, excited to, to learn more about your work. Maybe before we go, are those, um, applications that the students built? Are they available anywhere? Are they open to be showcased or viewed by outside of students,

Simone Sessolo (26:34):
Sir, that was the, uh, intent to make them, uh, to, to give them to the organizations. And then the organizations could use them as they see fit, but then, um, you know, um, everything happened. And so, uh, we had paused on that. Um, I I’m, because of proprietary respect, I feel a little, um, shy in, in opening them up to free access. But of course, uh, if you, if you want it to see how they are, I can definitely, um, share them to you. Okay. With you. Yeah.

Jeremy Nelson (27:14):
Yeah.

Jeremy Nelson (27:15):
Okay. And, and we’re talking about doing an XR summit next spring and having like a student showcase. So, you know, I’ll be sure to let you know, if you want to let those students know they can showcase their work.

Simone Sessolo (27:25):
Yes, absolutely.

Jeremy Nelson (27:26):
At that. So. Okay. Well, great. Well, thank you so much. Simone, it’s been a joy talking with you and I really appreciate everything.

Simone Sessolo (27:33):
My pleasure. Thank you, Jeremy.

Jeremy Nelson (27:48):
Thank you for joining us today. Our vision for the XR initiative is to enable education at scale that is hyper contextualized using XR tools and experiences. Please subscribe to our podcast and check out more about our work at https://ait.umich.edu/XR.