Welcome to the FuturePerfect Podcast where we talk with compelling people breaking new ground in art, media, and entertainment. This podcast is produced by FuturePerfect Studio, an extended reality studio creating immersive experiences for global audiences. Episodes are released every two weeks, visit our website futureperfect.studio for more details.
The text version of this interview has been edited for length and clarity. Find the full audio version above or in your favorite podcast app.
For episode 005, Wayne Ashley interviews Nick Fortugno, co-founder of the New York-based game studio Playmatics and designer of numerous digital and non-digital projects, including board games, collectible card games, large-scale social games, and theater.
INTRODUCTION AND ROLEPLAYING
Hey Nick, thanks for joining us. I’m really excited to dig into some of your background, ideas, projects, and particularly your alternative vision for a future of theater. I see you as a catalyst, a kind of cultural interlocutor making links across different forms of knowledge and practice, and the work you've done really attests to this. You've designed video and board games as well as outdoor public games. You're the co-founder of Playmatics, a New York game studio and the lead designer on many theater works, including Frankenstein AI and The Raven. And of course, one of the lead creators of the blockbuster mobile game Diner Dash.
But first I want to go back a bit. Your cousin introduced you to roleplaying when you were quite young and you ran your first game of Dungeons and Dragons at six years old. Is it too much to assume that roleplaying is one of the most critical activities for you, if not a central organizing practice leaking into everything you do? Give us a sense of how roleplaying has activated much of your thinking and practice.
Nick Fortugno: I think a central organizing principle is like a good way of thinking about it. It doesn't inform all of my work in a literal sense, but it's the heart of how I think about aesthetics. In Dungeons and Dragons, essentially what you do is you tell stories with other people and you use a rule system to adjudicate disagreement. You have a lot of “I hit you”—“no you didn’t” stuff in roleplaying so you need rules to deal with that.
When you’re storytelling in that system and you're the person responsible for making the story, you don't story-tell the way you do in other forms where you have an idea of the story in your head and you're figuring out how to implement it in a way that will affect the audience. Instead, the players or the protagonists are interacting with you and they're changing it constantly. And so you don't know where the story is going. You have ideas of where you could go, you have ideas of what you might want to happen, but you're really in this collaborative process.
And so this idea of improvising and using systems to generate things and being responsive to the interactions of other people is very much at the heart of my work. It's how I teach, how I think about storytelling centrally, and it informs a lot of my aesthetics. So yeah I would not be the person I was today if my cousin Joey didn't teach me D&D (Dungeons and Dragons) in Glen Ellyn, Illinois.
DESIGN THINKING
You're also a prolific researcher, not only of games, but of literature, theme parks, new technologies, and performance. I'm thinking about a previous discussion we had where in one breath you mentioned cultural forms that most people would never bring together in the same conversation. The list is long, but indulge me here: the British theater company Gob Squad, Galaxy’s Edge at Disneyland, Harry Potter hotel, the theater collective The Wooster Group, the blockbuster event Sleep No More, the novels of Joyce and Pynchon, Evermore Park in Utah, and the epic video game Elden Ring.
This cluster excites me because it’s how we think as well, across these kinds of groupings. You also use this concept of affordances to enable you to think systematically across all these activities. Can you say more about that?
NF: Affordance is a concept from design thinking, Donald Norman really popularized it. It's the idea that a form has features about it that lead to certain kinds of use. There are things that are intuitive in a way, or natural in a way, that come from a form. If I put a handle in a certain place, you hold the handle and that changes your use of the device.
That idea that the forms start speaking to certain kinds of use cases is very central to thinking about interactive design. Because when you're a designer in those spaces you make the affordances. You don't tell users what to do. You give them something and you have them do it. That's why it's interactive. It’s not like a roller coaster where I strap myself in and I just ride the rails that were put out in front of me. It's more like a theme park where there's just a bunch of stuff. But I don’t go wandering off into the most boring part of the theme park. I go towards the lights, I go towards the sound, I go towards the interactive things. The design of those things that attract me, the things that challenge me, the obstacles and the rewards, all of that stuff moves me around in those spaces. This is central to the way I think about my practice.
LITERATURE, PLAY AND AMBIGUITY
You have a BA in graduate study and literature. In our previous conversation, you noted an overlapping relationship between post-war American literature and the kinds of interactive narratives found in gaming. Do I have that right?
In our other podcasts I've been really interested in what brings disparate people to these emerging hybrid media spaces. They come from film, dance, theater, visual art, and gaming. I think you're the first person in our podcast series making connections between Pynchon and James Joyce with interactive gaming structures. I'm curious about how you came to make these connections.
NF: When I got interested in literature I was drawn to postwar postmodernist approaches to writing, like I’m thinking fifties, sixties, and seventies. But really you could stretch it from a Borgesian and Joycean and Steinean space up through the modern day. There's still authors like Ali Smith doing stuff like this. But when you look at like things like Pynchon and Nabokov in particular, their works start becoming a little bit obsessed with interpretation. Interpretation becomes the center of the novel. The novels become games about interpretations.
There are other authors in that space who are really breaking down the sense of what you're supposed to consume from the story because they are, in a meta way, thinking about the fact that you're interpreting them. Whether it's Crying of Lot 49 asking you to think about what communication systems are and then challenging you on how we interpret conspiracies. And that's also all over Foucault's Pendulum. Or a book like Lolita, which is basically laughing in your face about your attempts to understand it. Or Pale Fire for that matter, which I think is an even deeper experiment.
What you see over and over again is this idea that the novel is a game that the reader is playing with the novelist. It's not a puzzle. You're not going to get the answer out of it. That's not the point. And certainly postmodern poetry and people like Asbury would argue that if you got one meaning out of a poem, you didn't really read the poem anyway. The work becomes something that you as the audience have some ownership of because it is open to you and because it’s an ambiguous object that you have to work with. That’s what got me. I was already, just from roleplaying, very used to the idea that I participate in stories and that they come from this relationship with me and the text.
So I don't like talking about interactive narrative. I think that's a bad phrase because I I'm always interacting with story. That's not new, what's new is the types of affordances of interaction that I get from stories, and what the possibilities for changing those stories are, and how much the story is a fixed thing that I encounter, and how much the story is flexible to my input. To me, the literature study was partly just giving me an outlet for stories and a place where stories can actually be quite experimental because when you just write it's cheap to make crazy worlds. It's the same amount of ink to write a crazy world as it is to write a realistic one. You can go very far with literature in a way that would be harder to do in film because you have to shoot all that stuff. The drive of novels from the modernist period on has been a drive towards more and more stylistic experimentation and that has been really engaging to me because you start seeing it as almost a formal thing. You can look at it like a structure and then you can see that the structure is doing something.
Joyce’s Ulysses is an excellent example of that. Each chapter is written stylistically and formally different. There are chapters that are dialogues, there are chapters where the stream of consciousness changes radically, there are chapters that drift, and that's part of the narrative. If you go back to the Oulipo experimentation that Calvino and other French and Italian authors were doing, they were literally creating that whole idea of branching trees. You start to see that there are patterns of structures of story that we can start to establish.
That's the approach I take to this question of rhetoric. Exploration is a set of tropes, and branching is a set of tropes. It's similar, whether you're branching in a YouTube video or branching in a choose your own adventure, or branching in a game like Until Dawn. The branching is similar, it has similar tropes. So we can look at it structurally and say, well, what does the structure do? How do the choices in the design of the structure change things independent of content. And then what is the intersection between the content and the structure?
DYNAMIC STRUCTURES AND GAMES
It's interesting to note how the strategies found in avant-garde and experimental literature have leaked into, or have become one of the dominant ways of constructing narrative within popular culture, video games, and even marketing. What was on the periphery has, in a sense, moved to the center and become part of the entertainment industry.
NF: I think so because as you start moving into more dynamic and particularly digitally dynamic work it starts to have to be structural. Although that spills back into the analog, especially as internet of things (IOT) becomes very reduced in size and cost and technology starts coming back into the real world. You start seeing this there too.
I'm riffing a lot on arguments in a book called Expressive Processing by Noah Wardrip-Fruin. If I make a piece of work that changes with every user and produces a different outcome, then the output of that work is not really an analysis of that work. If the work has a hundred thousand possibilities, one possibility is such a small segment of what it could be. That it gives me information as a user, but I can't really critique the work from that perspective. I have to look at the structure because it's procedural, it's not predetermined. And I think as we start moving into works that are like that, and since computers enable us to do that, that's what computers are good at is that kind of dynamic procedural, then we start to see that structural analysis and system design become more and more important. As it does, and we see the affordances that has, we can start pulling those affordances into other forms where we see similar audience relationships.
So I don't think: does theater need this? Does film need this? Does installation need this? No, It doesn't need it. You can make good art without it, and obviously we have made thousands of years of good art without it, but the possibilities of the art change when you start seeing those things. That's why I think it’s starting to permeate. Digital games are a very big industry and there's been a lot of really interesting storytelling in them. I don't think all people who study this stuff know that because it's locked a bit behind barriers of picking up a PlayStation 4 controller and trying to get through it. Shadow of Colossus, for example, is one of the most important digital works ever made. But not many people experience it because it's a really hard digital game. And it has to be hard. That's part of its aesthetic.
But I think that the people who have bridged this are starting to see that you can inherit things from those forms into these other spaces. That's just changing the way we think and then you start to see work in the world that is just more procedural. Work that does just become more dynamic in its nature. Then you end up with stuff like LARP (Live action role-playing) where, you can't make LARP the way you make theater because I don't know what the players are gonna do. So my scripts in LARP can't be like a theater script, it doesn't make sense. I need a structure that will support 40 people running around doing random things.
PARTICIPATORY EXPERIENCES DRIVEN BY TECHNOLOGY
This brings me to theater, particularly two participatory theatrical installations that you co-created. First, Frankenstein AI: a monster made by many which was an AI powered immersive experience that premiered at the 2018 Sundance Film Festival. And The Raven, which was performed as part of the Lincoln Center’s New York Film Festival in 2019. Tell us what audiences might have experienced when they participated in Frankenstein AI and what was the genesis of that work?
NF: Frankenstein AI has had a couple of different forms. Its original form was a small audience immersive experience where you came into a room and you interacted with another audience member at a surface computer that was like built into a table. It was formulated as an artificial intelligence asking you questions about what it was like to be human and you're sort of marking values on the table using a physical computing device that looked like an ouija board. That information was sent to an actual AI that was in a cloud which was used as the seed to determine a mood that the AI had. And then when you finished that exercise, you were brought into a room that was mapped with projections and IOT procedurally played drums and you would have a chance to talk to the artificial intelligence.
The artificial intelligence would generate a question and then it would be delivered in text to speech to the audience in the room. And then the audience in the room would direct the docent to type a question into a typewriter and that would be sent back to the AI. This was all formulated where there’s this AI that's been created, it has escaped into the internet and it is trying to understand what it is and what humanity is. And it’s using the narrative of Frankenstein as this thing that was created that doesn't understand its role as a seed to understand where it's going.
The whole thing was essentially a meditation on two things. One is this question of what is AI and what should we be worried about AI? These were the conversations that I had with Lance Weiler and Rachel Eve Ginsburg who were the co-creators of that project. My big argument was that everyone worries about Terminator, but what we should really be worried about is Kafka. AI is not a monster that takes us over. AI is a thing that doesn't understand us and then just acts procedurally in ways we don't understand.
This is around the time that Microsoft had released an AI that became wildly racist and we were thinking about what it meant that we're teaching AI and how could we make a piece that gets people to reflect on the idea that we're engaged with artificial intelligence in the world? We are training it and we are going to teach the AI what it does. So if that's the case, what is our responsibility? The whole piece was kind of a meditation on that process. I did the creative technology design on that and some of the interactive narrative design of the sequencing of it. I'm very proud of that piece personally, because it was the first piece of creative technology that I ever actually showed in an exhibit.
I worked on the technology that connected all of devices. So it meant that when the AI changed mood, the projections changed, and the drums changed and it pulled the AI's response and then fed that into the speech to text and delivered it into the room. So I basically did the technology that connected the surface tables to the AI, to the projectors, and to the drums. This was a topic of research I've had for a long time about how technology could be used to create these like kind of seamless connections between things. You didn't see anything happen, you just asked a question and suddenly the projections and drums changed. I call that seamless technology—technology that doesn't have clear lines where it connects. I think that could be a kind of magic and that was important to me.
What did you learn from producing Frankenstein AI that changed your approaches when you then began to develop The Raven? How does The Raven work as an experience that grew from or built upon your previous work?
NF: The Raven was an immersive performance where we allowed an audience into The American Irish Historical Society where they experienced a magically real story of Edgar Allan Poe’s The Raven. The center of the technology of the piece was that every user had a lantern that they carried around with them. The lantern was an IOT device that was reading beacons in the space and connected to a central system. The audience also had a set of headphones that were playing audio for them. So most of the audio that was present in the piece came from the headset that was being played based on where they were and based on a character they picked at the beginning of the piece. Everyone was sort of playing a performer in the piece. The performer Ava Lee Scott, who was playing Poe and co-wrote the piece, was moving through the space as Poe meditating with these characters. But you, as the audience, were one of the people that Poe knew from his life or his creations.
What Lance Weiler and I carried from Frankenstein AI was this idea that we could create a central technology system that was guiding all these users without having to have actors on top of those users moving them around. And that the storytelling could really be based on their decisions, because it was in part based on where you went and what you encountered.
The other thing that Frankenstein AI taught me, in a real sense, was that these technologies could be stable. The work had a server system, that's how it ran, it was a server that was running on a small piece of technology called the Raspberry Pi. We turned it on and on the first day when we were running it we just didn't turn it off. We wanted to see if it would stay up overnight. And then we didn't turn it off for two full weeks. It just ran nonstop for two weeks and it never broke. We never had to restart it. So that taught me these things can be made battle ready.
We brought a similar kind of technology to The Raven. There were obviously different technical constraints to The Raven and there were different bugs we were facing, but we went through a similar process of creating a central system that guided the narrative. If we do that right and we have the right affordances to connect to the audience that can take the place of a bunch of docents, a bunch of rules, a bunch of structures, and people can just explore. Then through that exploration they can find story.
I should say that we worked with pretty robust technologies on that project. We were in partnership with Microsoft and we were using pretty heavy Azure servers and things like that, but it was not for heavy lifting stuff. It was for reliability of the delivery of the material. And then we built this gigantic XML file that was the branching script of the entire piece so that we knew where people were. We could time lights and sound cues and things like that.
THE LIMITS OF THEATER
What I find compelling about both of these projects is their capacity to posit alternative models for theater's future. They either directly or implicitly suggest that theater needs to be remediated or fixed. For the purposes of this discussion, can I make that assertion?
NF: Yeah, I will also defend traditional theater, but… [laughs]
That’s good [laughs], but what is it about certain kinds of theater that need to be remediated and how are your explorations accomplishing this? I’m very careful to say alternative models and I’m not asking you to generalize. I think from our audience's perspective, people are going to ask: what’s wrong with the kind of theater that I do? And why do I need these other systems? Why do I need to even consider these technologies? All these kinds of questions are implied, for better or worse, in the kind of work that you're proposing and the kind of exciting research that you're carrying out.
NF: First of all, there's just aesthetic possibilities that are very hard to create in a linear format like theater. Guilt is hard to create in an audience. Triumph is hard to create in an audience because they don't do anything. You can get to shame, but there's types of shame you can't get to. So there's aesthetics that become possible just when someone is culpable and when someone has the ability to achieve. That becomes kind of interesting.
Games have lots of emotions attached to victory and failure that can be leveraged in all sorts of interesting and weird ways. There are pieces like The Privilege of Escape, which was an escape room that was a meditation on systemic bias. That's an interesting example of a piece where the designer was trying to use the affordances of games to demonstrate a problem in the world. And games typically do that. There's just pure emotions that are inaccessible to linear media. I think because there aren't affordances for the audience to access them, despite the diversity of emotions that these forms can create.
The second possibility is, it’s a question of how you want to engage with your audience. As an artist, I don't really like telling people stories, that doesn't really engage me.
You're the second person we've interviewed who has talked disparagingly about stories and storytelling. Say more about that.
NF: I don't mind being blunt about this. I'm not that interested in my biology. I'm not that interested in my history. I don't find those things that interesting. I don't think I have a vision of storytelling that's so powerful that some muse came to me uniquely and now the word of heaven is coming through my body or something. And this isn’t to knock people who do that, there are geniuses who make that work, but that's not how I create and that's not what I do.
What I want is to play with you. I want to be able to engage with you and you know, catch the ball you throw and throw it back. And this isn't altruistic just to be really clear, I mean I like doing that with people, but it's also really fun to catch a bunch of balls coming at you in crazy directions and keep the whole thing on track. There's an artistry to that. That's what running an RPG is, it's like throwing track in front of a moving train. So I think that's really powerful and you get things that you would never get otherwise.
Similarly, if you jam you get something that you would never get when you compose. The improvisation and the participation of other people leads you to create something new and you can do that with audiences. And you can do that with audiences in ways that don't make crappy, thin, gray, over-democratized work. Because I'm not saying that's not a problem, if you just let everybody come in and cook in the kitchen then you get no food or you get bland food or inedible stuff.
Structures make it possible for people to participate in ways that are meaningful, but controlled, that fit within the aesthetic. So people understand what kinds of creations are possible in this space. And that is a whole set of techniques that then allows audiences to come in completely ignorant of what you're doing and then tell a story that they helped make that is still in the aesthetic you wanted. There's a magic to that that I think is really powerful. It opens up whole new kinds of forms and it's a different way of engaging with the world for the audience and I think that's powerful because we haven't really seen it before.
There are some experiences like that, but they tend to be very high demand on the creativity or they tend to be gate-kept or they’re high skill-based. And what immersive theater can do that I think is unique and independent of digital games and LARPs, is that they can be approachable. I can show up and not really know much and still participate. And I think that's a space that's really powerful.
And then the third beat that I just have to mention all the time is that tickets are very expensive to these things. They charge a lot of money to get people into those things. I think that there's opportunity, from a business perspective, if you can figure out the scaling. You're seeing pieces like Particle Ink in Las Vegas which is a piece with projection mapping and dance where they're starting to figure out how to grow the audiences in ways that don't hurt the piece. You start looking at genuine business models for keeping those things up. What are other business models that can keep dancers, actors, and set designers involved? Because none of those people are going away in immersive theater, we need all of those people. We need them the same way we need them in other forms. It’s a parallel skill if not an identical skill right. So we're not telling actors they're out of work. We had actors in The Raven, the actor was the center of The Raven in a lot of ways, but the actor was supplemented by all of these other things to create a new form where people can explore and make choices and feel directly engaged.
NEW FORMS OF PEDAGOGY
Given this technologically seamless environment within which performance might take place, do you see the training of actors taking a different path? Or different ways for how writers produce scripts? Do we need new kinds of training for scenographers, sound and lighting designers that will accommodate and respond to these ideas and new approaches to performance?
NF: Well acting, for example, in these kinds of cases, has a lot more improvisation in it. It's much more deeply based in that kind of improvisation, but it's also a lot about vulnerability. This is something that I'm just going to riff off of a writer and actor that I know Char Simpson would talk about. Char was part of the Blackout Haunted House for many years and talks very much about how they created vulnerability and that the creation of vulnerability was really important. That becomes a different way of thinking about acting. But also the idea that an audience member might ask you your favorite color and you need an answer that seems natural. That's a more roleplaying kind of acting than I think some actors are trained in, of course some actors are good at that. You don't know what's going to happen so you can't write a script the way you would normally write a script. It has to have some variation in it. You have to think about it more like story, like world building.
I think directing changes because I don't know when we're gonna hit a specific moment or I don't know what perspective I'm gonna be coming from in a specific moment. So I have to think differently about that too. And you see that in digital games which will sometimes have cut scenes that are very film-like, but they'll also have scenes where users can walk around and watch what's happening. Which is why when we talk about VR we talk more about immersive theater because the viewpoint is not singular, it is a multiple viewpoint environment. So I'm thinking about it more from that perspective. Theater in the round is also relevant here. Again, that's not a new form, but it solved this problem. So maybe VR should look at theater in the round and then learn some lessons for how you keep an audience's attention in a broad space.
And in fact, we're getting that big, we could think about station-based theater where people are really just drifting over a whole plaza and engaged in an experience. Are these forms going to change acting, writing, directing and set design? Sure, of course they are because the affordances of the audience are going be different and that's going to lead to different outputs. But it's not like we made up all this stuff just because the technology came along. We had happenings, we had station-based theater, we had rituals.
I'm thinking about the Ramlila which I participated in India many decades ago in Varanasi. This is a month-long event that is played out over the entire city in which the inhabitants take on all the various roles. The city performs and becomes an immersive ritual and religious space. So there are absolutely precedences that are centuries old that we can draw upon. I’m thinking about how the pedagogical needs of theater will continue to change in response to these new forms that are becoming more and more central to our lives.
NF: Yeah I teach immersive and dynamic narrative and I teach it in the way that we’ve been talking about. I teach it in this very broad, cut-across-media way. Media does not matter for the purpose of the class, that's not what it's about. It's about the tropes that the media use and how those things relate. And then you see this in disciplines like narratology where people are really coming at narrative from lots of different directions and trying to figure out how stories get told.
Another point that's just very important to me is in the intersection of these forms. Because you're not going to get immersive theater from theater alone. There's a bunch of pieces that theater doesn't really know about like interaction design and a sort of multiple viewpoint about the pacing for that kind of stuff. Games understand that, but games don't understand what theater's good at. Games don’t understand how you create scenes or understand how you create dramatic power, and games don't understand the value of liveness, frankly. Some of that we can get from LARPs, but LARPs aren't theater either. So it really is in the intersection of all of these fields.
I think more of this is happening. You're seeing escape rooms get more theatrical. I think it's too slow, like way too slow. We could have gotten to where we are five years ago and we could be five years ahead of where we are right now. But you're starting to see some of that thinking happen. You're starting to see immersive pieces that are bringing some game elements into them. You can have conversations with people about VR where you talk about digital games and they don't scoff.
This focuses again on the ideas of interaction and affordance and how those relate to storytelling that changes the orbit of everything. And then the skills that people have been learning, like the acting, writing, directing, set design, costuming, they all have a place. They're all going to be there, they're just going to circle around a different sun. And that sun is this audience member who can change what you do. That's different.
Nick, thanks for all of the conversations we've had. I look forward to working with you. I think you're a really important thinker and maker, and your experiments and research bring a lot of insight into the future of performance.
NF: Thank you, I appreciate that there are people like you that are thinking about these problems and working in these problems. Like with your own wonderful work and that podcasts like this exist to have these conversations. I look forward to a really bright future because there's other people like you in it.
#005 - Nick Fortugno: Storytelling, Design Strategies, and Interactive Theater