Video Games and AI: The Future of Storytelling

Announcer:

Today on Building the Open Metaverse.

John Gaeta:

If you saw an X-wing fighter roar overhead during some story moment, you could actually click on it, and you'd be in the driver's seat, and you would be taken down into a war scene; then you could basically play and engage.

Announcer:

Welcome to Building the Open Metaverse, where technology experts discuss how the community is building the open metaverse together, hosted by Patrick Cozzi from Cesium and Marc Petit from Epic Games.

Marc Petit:

My name is Marc, I'm from Epic Games, and my co-host is Patrick Cozzi from Cesium. Patrick, how are you?

Patrick Cozzi:

Hi, Marc. I'm doing fantastic. We're in for a real treat today. I'm looking forward to learning a lot.

Marc Petit:

Yeah, because today, we're super happy to welcome John Gaeta.

He's an inventor and innovator, an academy award-winning designer. He was Senior VP of Creative Strategy at Magic Leap, Chief Creative Officer of Inworld AI and works on a number of things we're going to talk about for the podcast.

John, we're very happy to have you on board. Welcome to the show.

John Gaeta:

Thanks, Marc and Patrick. Appreciate it.

Marc Petit:

So, the first thing we'd like to know in the podcast is to ask about your journey to the metaverse, and in your case, immersive storytelling has been at the heart of everything you've been doing; it's probably a long journey, but please introduce us to your background.

John Gaeta:

Yes. I think everyone's journey is probably not necessarily a deliberate one. It's a lot of serendipity, and I'll try to be brief about it.

The short story is I chose to go to university in New York City, went to film school there at NYU, had that life experience, began understanding the limitlessness of storytelling from mentors, teachers, and such, and then began experimenting.

But of course, the real sandbox for someone doing that is just living in New York City itself because it is just so vast with things you would never expect to be happening on the other side of a door. I met a lot of people and wound up finding my crowd.

I lived in the East Greenwich Village for a while, met a lot of artists that did a lot of things that had nothing to do with film, and eventually found a home relatively quickly working in animation and stop-motion animation of all things. At the time, it wasn't really a very regimented area of filmmaking. I mean, there wasn’t a heavy union presence in those times because it was just so unusual in just this routine that one would have to go through.

I would essentially babysit Mitchell cameras until 5:00 in the morning almost every night of the week while animators hand-animated and pushed their little figurines up hills night after night. Because I fell in with that crowd. I was lucky enough to hear about some unusual happening that wasn't in New York City.

It was the fact that a fellow named Doug Trumbull, who passed away last year, was retreating from Hollywood and starting some super secretive high-tech studio of some sort in Western Massachusetts. It truly was actually what he decided to do there was to essentially push the bleeding edge of everything you could imagine, from robotic camera systems to automation, and how miniatures and things like that were made.

He embraced things like using CAD, and I think maybe this is where we have our first connection. I mean, we experimented with using software like Softimage, for example, for experimental camera path creation.

Eventually, we realized that there was a lot more to be done once one understood essentially how a physical camera or how we physically could relate to some digital content. It seems very simplistic these days. People take it for granted, but just the very ability to graft a computer generated... Back then, it was the early days. This was just not that long after Tron, had a graph to a physical thing was a really big deal.

For me, it unlocked some thoughts right off the bat, how the real world could intertwine if one understood some essential things like light. How does light fall upon both equally convincingly? Of course, spatially, how do they interlock, and eventually, dynamically, how does something in motion in the real world interlock with a generated thing?

Even from those early days, back then, we didn't really call it computer vision quite yet, but it was early days for that. To me, it was like paintbrushes and all of that under Doug Trumbull's super imagination and high-tech factory. He was the type of person that he was like a toy maker, but with high-tech toys.

If something didn't exist, like a camera that could shoot IMAX but fit inside a space this big, he would make it. He would machine it, make it, and stick it on a robotic camera that he also had made. You're in this atmosphere where if there's a limit, you can move past the limit; there were some pretty interesting lessons there. That begins a journey, I guess.

I've gone from those times to always believing that we could get deeper into these things and kept hanging with colleagues that were supercharged. I did meet a few that, to this day, they're still my friends to this very day.

That is another benefit that you don't have to be contained and aligned to one company. You can have colleagues that move and migrate around into really fascinating places. If you just stayed friends and you keep speaking to each other almost collegiately, you can learn quite a lot about what's happening and what's changing. I guess that's why we're still here talking.

Marc Petit:

Well, you can even go pick up an Oscar on stage with those friends.

John Gaeta:

The amazing thing about being in the game this long, at least a couple of decades or more, is that a lot of us began reading about science fiction in scripts that were in things in movies that we've been in this game long enough to see those things manifest to actualize, to actually help in actualizing these things.

Marc Petit:

I alluded to The Matrix, of course. We have to talk about The Matrix and your role as a designer, and all the work you did with the Wachowski siblings.

You did win Best Visual Effects Oscar for it at the 73rd Academy Awards. Was it back in 2000, right?

John Gaeta:

For certain, it felt interesting because getting deep into the '90s for the general public, for society at large, was a big decade of change because people began using and feeling comfortable with the internet. I'd say at the beginning of that decade, it wasn't so.

It was a strange, unnatural thing, I suppose, using your computer all the time to interact and communicate. But by the end of that decade, people were doing it, and of course, it would just exponentiate the following decade.

With that became the capacity of the everyday person to expand their thought about what was possible on the other side of that screen, I think. It was relevant. I mean, it was stylish as well for the times. It was edgy for the times; the thing that I think resonated was this idea that we're heading towards some digital life that could end up essentially being, for some people, a way of life.

Patrick Cozzi:

John, beyond movies, you started looking at game development and immersion tech.

You were co-founder of Lucas Films, ILMxLAB, and you're SVP of Creative Strategy of Magic Leap, Executive Creative Producer of The Matrix Awakens, and now Chief Creative Officer at Inworld AI. I mean, what was it about video games and immersion tech that brought you in?

John Gaeta:

I will always feel blessed to have had as many years working with storytellers, writers, and artists, and filmmakers just to understand how to sculpt a story and how powerful, of course, that can be and rewarding that can be, but the one thing about the movie business, which was really it weighed heavily, was to make a movie is a mountainous effort; it still is a mountainous effort. It takes a lot of people. It's a really big difficult product to make and even more so to make an exceptional story and film.

The thing that was always difficult about the end was, well, if you lived for the end and the film didn't turn out to be a masterpiece or the audience just didn't connect to it, it doesn't feel great. Two things came from that feeling.

First, I, myself personally, had to get more Zen about the process along the way. I mean, it is, of course, enjoyable, but you can't really get fixated by the end, necessarily.

Everything that you do, you do it the same way. You give everything you have and hope for the best. But if you're not getting something rewarding out of the whole journey in some fashion, as hard as it can be, if you're not passionate, then there's a problem there.

For some painters, it's only about the act of painting. Then once the painting is done, it's dead to them. So, the first lesson is that I had to learn to be okay with whatever happened at the end, but the second part that was really just getting at me after a while was the world seemed to end when the movie ended.

You went through all of this to world build, to make a universe, and then the movie would end, the lights would go out, and you'd go home. The world basically stopped at that moment. That really was difficult for me for a while. It really started to motivate me to start thinking about dynamic ever-living worlds.

Real-time really was an opportunity in games, and all of the technology of games was an opportunity to help build a living world. That was a big motivation. Of course, it wasn't so easy. It's taken quite a while to try to get a persistent world through games and other real-time entertainment, immersive entertainment. We're still pretty much in the early stages of that thing, but that was my main motivation.

I thought that if it was the type of world where you could either bear witness to or literally be a part, by way of your journey through it, your path through it essentially creates as rich a story as one could have in some of these cinematic experiences, well then you had it all right. You could have in and of itself an infinite matrix of individual stories crisscrossing through an interactive world.

That could be through gameplay, it could be through other kinds of engagements, but there's really quite a lot of potential there, and it's just only going to get better.

Patrick Cozzi:

I wanted to double back on The Matrix. I mean, it had a huge cultural impact. I still remember the first time I saw it, and I'm sure that's true for many of us. Did you think that would happen?

John Gaeta:

No, of course not. I mean, no, you don't really know these things.

That's what I was referring to before was that if you've made the choice to commit yourself to go into a project of big film or game project, you have to put everything into that. It's going to be a big investment in your life and career, but you just don't know whether the audience will take it up.

I suppose it's different now once the world is up and running. If there's engagement, you can predict how the participants and fans will feel about it, but back then, no, we had no idea.

Marc Petit:

A lot of new technologies were pioneers in those movies. Bullet time is well known, but it was the early use of photogrammetry as well for real-time movie making and a lot of work on digital humans as well.

John Gaeta:

Very little of that would've happened without, quite frankly, the participation of individuals. There are a number of individuals in Epic that were in on those projects. Essentially, we had the freedom to pursue whatever techniques we wanted to, but also a bit of a thesis.

We were excited about this project, and we decided that we would do everything we could to imagine what methods might actually be needed to make virtual reality.

That immediately steered us towards less about fabricating things from the ground up and more about capturing reality capture. Our thesis was to do everything we could to get the first building blocks of reality capture up and running. We ran with that, and we were backed by...

The Wachowskis were like, "Do it. Go ahead. We'll shoot back up shots for you in case you fail," which was smart, but they allowed us. I mean, really, quite a lot of people. No, I can't get into that now, but that was the thesis.

From those days to today, you see some of those things. They went through years and years of a lot of mimicry of it, but you absolutely see some of the pillars of volumetric capture, for example. You can trace back to some of those experiments, image-based maps, and three-dimensional maps.

There's like crazy things that you could... I wouldn't say it all began with The Matrix because we ourselves were inspired by little kernels around the world, but it was those times.

Patrick Cozzi:

We also love that original transmedia vision for The Matrix, where there was the movie, but there's also The Animatrix series and the video game.

Was there a comprehensive strategy behind that?

John Gaeta:

That wasn't by design by any studio. That was driven by Wachowski passion. I mean, they truly were influenced and super energized by comic books and anime. We know all the lore at this point about them, but it was true. They played video games. They loved all these things. They were really living in the times as younger men. It wasn't just that they wanted to make a big franchise.

One could debate its success. There certainly have been other attempts at transmedia franchises like Star Wars, for example, has had more success for certain, but the idea that what they really wanted to do was to have the stories cross over one another and to allow the fans to expand their ideas of the universe through all these different channels.

It was an instinctual thing for them to do. It was an instinctual strategy. At the time, in Hollywood, they had the ability, the power to push into other products. So, they tried it.

Like I said, I mean, we made an MMO. We did all these things that weren't really common at the time. The Animatrix, also rather novel at the time to essentially not just... How would I put it? Hollywood's really good at ripping off other brilliant creatives and works from the past. We knew that Japanese and Eastern animation was an absolutely critical influence on our visuals and choreography.

Even the way events happen, inexplicable events, we were looking at that, and we're like, "Wow, they are really ahead. In many ways, they're freer."

We wanted to honor, to some degree, as opposed to just rip them off. We wanted to honor them. We allowed them, a number of companies and great directors, to do these shorts. It was really a great meeting of two things.

Since then, a number of properties have tried this again. But the idea of The Animatrix and that other way of telling stories, I mean, it's something that's these days done very well through stream air channels and such, but back then, it was a pretty new feeling.

Marc Petit:

Then 20 years later, The Matrix Resurrections, how do you come to pass? Because 20 years after, the stakes were much higher, people had a better understanding of virtual worlds, immersive, and metaverse had become a household word.

How did that impact how the team approached the movie?

John Gaeta:

It feels like a movie about people who were together at school or college, then they all went to have separate lives, and then they come back again unexpectedly, a reunion. Oh, we're going to go to the beach house and hang out together.

We learned that everybody is way down the road on things. It's interesting in that regard. I mean, Kim and I, Kim Libreri, who's a CTO of Epic, we were at this point trying to actualize some stuff. We're trying to make real-time worlds, and we're not sure immediately.

Well, first of all, it was wonderful to, essentially as friends, because we spent years together when we made The Matrix. We spent years together; we went through a lot of stuff together, making those movies. 9/11 happened while we were making those movies.

They were a pretty bonding experience for everybody. Then we all left and started doing different things, but Kim and I were like, "Okay, we're heavily involved in other stuff now. I don't think we can circle back to visual effects right now."

They shot the picture during the pandemic, which made it even more crazy to pull off. But to depict what it would be like to be in a cinematic fidelity open world that you could play or tell stories in was really where we always wanted to be going anyhow and what we've been chipping away at for years on end. We're like, "This is how we'd like to participate." So, we did. We worked with the actors and with Lana and others on that. We were a little bit cautious on the cinema.

We were going to update as you see fit in the way you see fit, and we'll just find our happy space in that endeavor. I also thought that "Well, let's continue some other aspects of the thesis." Not a lot of people know this, but there was a little bit of a larger plan. Here it is.

I'm going to come back to I had hope that somehow putting this universe back out there in our times in 202X, the emerging AI age and all that's going on with the metaverse, et cetera, it would be natural to be reignited. I was making certain assumptions. I thought, "Okay, I'll prepare for this. I'll see how people could get deep into this, deeper into this, assuming that this happened on a social level."

My thought was, well, it wasn't just Awakens as a demonstration, but it was like what was going on there was also this thought that it could essentially be the back lot for a new form of experiential storytelling that we could make new Animatrix and immersive byproducts of all sorts.

I started down this road testing for this and writing up some papers on this and doing some tests. We did other tests that were Love, Death, and Robots on steroids, Ambition meets Matrix Awakens, et cetera.

Thank goodness that Epic had the will to complete that Awakens project because it's the one piece really that got all the way through.

Marc Petit:

In season two, we had a full episode on the making of Matrix Awakens. If people want to refer back to it more technically, they can look back at that episode in the podcast.

Let's switch back in time to another franchise where you got to practice immersive storytelling, which is the Star Wars franchise.

In 2013, you joined Lucasfilm to contribute to the renaissance of Star Wars right after the Disney acquisition, and you co-founded ILMxLAB, where you got to develop several immersive stories and design projects.

First, I have a personal question, I remember you on stage at the Softimage User Group in 1999. You were the anti establishment person, and then you joined Disney. What happened?

John Gaeta:

Throughout my career, I swing on the pendulum back and forth between who doesn't like being heavily resourced with incredible technology, et cetera, and incredibly talented people around you. Everybody likes that. But after a while, those institutions can really grind us all down, and we need to essentially have a moment where we can be free, not encumbered by bureaucracy, et cetera.

Sometimes you really want to get into a more self-governed startup mentality where you can really take high risks. Sometimes you can take high risks in institutions, but it doesn't last forever. With coming into Lucasfilm, of course, a lot of friends were there before I got there.

I was almost intimidated coming there with such legendary brilliance in there, but there was going to be a new generation of Star Wars, and there was going to be literally, for the foreseeable many years, all sorts of multimedia activity.

Kathy Kennedy, who was super excited at the outset, there’s going to be a bit of a renaissance period for them on every level, from concept to writing to world-building to high-end new methodologies for the next types of media that Star Wars could be put into.

I mean, Star Wars, in the past, when it was really doing very well, was always a franchise that you would see the next generation of something like video games, all sorts of things.

At the time, just before I got invited to come in there, I was experimenting with this stuff with Microsoft. I was experimenting with augmented reality back in 2008, 2009, 2010, 2011. They had these with the beginning of these labs that they were using, things like the Kinect, but not for overly simplistic button pushing, but for basically augmented reality experimentation, amazing experimentation.

I was doing that and hanging out in the middle of San Francisco and the city at one of these stealth coasts doing that stuff. Because LucasFilm was going to embark on all this new innovation across years, they're like, "Oh, if you want to consider coming in, this is a good time. We're willing to take risks and take chances."

It really was that atmosphere, so call it a renaissance moment. Then did, in fact, go in there and did, in fact, get the freedom to play around with different things they had.

Again, this is another intersection with Kim Libreri. At the time, he had reconstituted the end of Lucas Arts into probably the most advanced real-time graphics lab that existed at the moment, really insane stuff, stuff that even today would be really impressive to people. They were playing around with this thing called Lucas Live, which was the building blocks of the way that virtual production stuff and Mandalorian approaches and such. They allowed me at the same time to be with one group that was a pure story department.

This is Rick Carter, the production designer with Doug Chang and the best-in-class of their concept artist conceiving of things that could happen in Star Wars that hadn't been written yet. It was a huge, massive art moment, just painting, painting, painting, stream of consciousness church of Rick Carter ideation, and then writers would come in, Kiri Hart. The head of development would bring writers in to be inspired by, and then they were writing scripts and looking at ideas at the same time.

It was pretty interesting that way. Then over on the other side, we're like, "Oh, well, let's see. What's the highest fidelity real-time that we could possibly do in case we could one day create essentially episodic live or otherwise real-time content?"

Other things that we got to play around with there… I mean, I started essentially this holographic cinema division.

We took the idea of big caves that were at the time only used for car design, and you would look at your car in stereo. Then we devised a method to actually, in a similar way, free the viewer to see that which was projected from any angle, any perspective they want, but instead of cars, we put full Star Wars scenes with characters that were live-driven by mocap actors; you could go in a full field of view of holographic scenes of Star Wars and have live interactions with characters.

To this day, it precedes. Still, there're no mixed reality glasses that can deliver a completely immersive, full field of view holographic. It was a preview, in a way. It was a way to preview what it would be like.

Derek did all these sorts of things, did early VR. We had the first headsets for Oculus. We put Bob Iger inside of a headset for the first time to see. We created these, we called them, immersive cinema tests. We did these. Okay, so this is fun.

We essentially created this universal media prototype. It was done with a modified game engine, and it was really, really beautifully rendered scenes that took place inside of a Star Wars scenario. There's a whole dramatic scene.

At first, the way we'd present this demonstration is you would see what you thought was an experimental scene in a forthcoming movie. It looked that good. You could have believed that it was a test scene from a forthcoming movie.

But then somewhere along the way, we'd be like, "Oh, we'll just like now drop the editing, and now you're there." You're now inside this dynamic real-time scene, and characters are still moving around; there’s like action going on in this live place. You, at that point, had the ability to just wander where you wanted to.

What you realized you could do was you could go to all of the... A lot of times with movies, you have parallel actions or, “Hey, this is going on while this is going on.” You cut between those things. So, with this demonstration, you could see everything that was going on that you had cut away from before.

You could be inside all of those different moments in totality, and then you could do other things in this demonstration.

If you saw an X-wing fighter roar overhead during some story moment, you could actually click on it, and you'd be in the driver's seat, and you would be taken down into a war scene. Then you could basically play and engage in a battle in a classic Star Wars battle. You could have this omni-media story if you want.

Marc Petit:

Yeah, we want that, John. We do want that.

John Gaeta:

It's relevant to what I care about now. Again, you can potentially see a day when you can have those things live in the same space and serve people in the way that they want to participate. Do they want to play? Do they want to be a supporting actor? Do they want to see someone else's story? But these things are now possible to converge, or we're heading that way.

Patrick Cozzi:

John, it's so cool to hear about all that innovation and that work there on AR and VR.

You also spent some time at Magic Leap. I was curious about what insights you might have gleaned there.

John Gaeta:

The strange thing is there's a lot of distortion, I think, out there in the blogosphere about Magic Leap. I don't know if it's resentment or what have you, but there was a lot of heat, as in a good heat. For a while, Magic Leap was a really interesting thing for a lot of people. A lot of extremely talented people from all places converged around this wild experiment to make mixed reality possible and spatial computing possible.

There were, I mean, yes, billions of dollars raised and spent, and lots of it spent on raw experimentation, lots of it, lots and lots and lots of experimentation. There was some distortion on the outside about, "Well, when will this day come when we are given this ability to be inside of our mixed reality metaverse?"

I think that is where a lot of confusion occurred with regard to... It's a very common thing to describe a destination like a road we're walking on and tour and do vision movies. All of the technology companies do these things, whether it's Microsoft or Apple or Google, or what have you. They create these destinations to inspire engineers and users, too, at some point, but hey, this is a place we want to go.

This future is possible. Come here and help us build it. For some reason, Magic Leap, when they tried the same thing, basically got charged with overpromising this future too fast, but I don't think it ever described the arrival time.

What was being described was the road that they wanted to be on. I'll put that out there.

Secondly, it wasn't their inspiration alone. I mean, there was a period of time, some years I was there, where the most remarkable humans on the planet making companies, building cities with grand visions for the next paradigm of computing would come through the door every day of the week to explain their vision, their interest; their interest in the potential of bringing computing into literally the spaces that we live and walk that we could, back to the beginning, not just stick computer graphics to the side of a wall, but have an understanding of what's going on here now.

To use that to essentially drive a simulation that you could actually put over the top, and it could adapt according to what's happening.

There was a lot of interest, not just there at Magic Leap, but coming from around the world. You could be a world-class educator, and you wanted to try to show a simulation of how to do surgery.

You could be an environmentalist, and you want to see how the glacial world is collapsing in the live simulations run over the top.

There're a lot of different use cases that people would come up with, and it was to the benefit of many to learn about those use cases, but that company got itself caught up in some difficult burn rate issues and lost probably one of the most promising groups of people, I think.

Those people dispersed into all the labs around the world that are benefiting from some of that near university-like education.

Nobody else has been able to really pull it off quite yet, either. I definitely anticipate and hope for a time that maybe Apple and others do present a nice next step, but it was actually two things at once. It was one of the most interesting experimental times I've ever gone through, including meeting people, people I never thought I would've met.

When I was in the movie business or back, let's say, in Lucasfilm before Magic Leap, I mean, to me, getting to meet Bob Iger was incredible, right? Here's an incredible person who's built the most phenomenal company you could imagine, right?

A multimedia wonder company based on character, blah, blah. I'm like, "Wow, that's intimidating to have a person like that come visit, although it was fine. He's a great, super casual person, comfortable to be around." I thought, "Wow, that's a big highlight in a career, getting a chance to meet and talk ideas with a person like that." Bob Iger would be that person, but at Magic Leap, incredible people would show up there every day of the week, mind-blowing from every walk of life, not just entertainment.

It was clear. It's the full spectrum that has an interest in this. So, I don't know. I have no regrets. You have this incredible time, and then you have the flip side as you watch probably the biggest unicorn of all time crash to the ground.

Marc Petit:

I'm not sure if it's the Mica demo.

John Gaeta:

Yes.

Marc Petit:

That was a big moment when Patrick, you could talk to a visual character in AR, and it was really an experience. Really, you get to go through to see what it was going to be to interact with digital humans and not know what to expect, but the quality of the interaction was super good back in the time.

John Gaeta:

That was John Monos’ special project, and he did quite an exquisite job with that.

Essentially micro-expressions, really incredibly super subtle capture, volumetric capture. But I couldn't even begin to explain it. I would highly recommend you talk to John Monos someday. He's an incredible person.

But yeah, I mean the dream at that time, again, Magic Leap imagined itself as the next version of Apple. It did. It had that crazed ambition.

Virtual humans, not just virtual humans, high fidelity virtual humans, highly expressive virtual humans, but also thinking virtual humans was a pet project over there. Sitting here today, I didn't realize how far we were back then. That wasn't that long, but it was fairly far compared to where we are today. That was the thought.

All of these things you think are around the corner, the metaverse, right? All of these things you think are around the corner. Technically, it might be possible to erect a vertical that can demonstrate the possibility of things, but then you're really faced with the market having no idea that it wants that, right?

You have to educate everybody. With mixed reality, there is a way to potentially do that, but I'm not hung up on mixed reality; it will come when it comes. But real-time worlds and immersive worlds are here today. They are really, I believe, the hub of many things. It's more like, "What window would you like to put on this world?" is where we're at with all the rest.

Patrick Cozzi:

John, we had the pleasure, a few episodes ago, Ben Grossman came on the show, and he gave an interesting perspective around interaction. He said for him, he just looks at his child, and they don't care about the super high fidelity content. They're just happy with interactive content that runs on their iPad.

Do you see it the same way?

John Gaeta:

Let's go back even further than kids with iPads. Let's go back to Walt Disney coming through your TV and telling you to use your imagination and provoking and prompting you to imagine beyond maybe something even you see, right?

The child's imagination, a child can have blocks, and they see worlds inside of their blocks. We have to measure what actually a child is receiving that way, I think, right? If anything, it could be that some of this digital media is limiting their imagination because it's basically defining it for them.

Let's just take that off the side. I mean, what your experience is about matters more than what it looks like. I'll say that. But the naturalness of realism actually… supports belief in fantasy and fiction.

I think it's equally important. I mean, it's two different subjects.

Do you need super high fidelity to have a compelling, interactive experience? No, you don't. Not at all. You can use the most primitive things to have a really interesting experience, but does high fidelity produce a form of belief that could resonate in a different way everlasting? Of course, it does. It absolutely does.

I definitely think that whether it's immersive worlds, game worlds that you see through a computer screen, or some VR goggles, or it's at some point the intertwining of simulation with the real world through mixed reality, I think all of those things are going to have huge impacts.

Patrick Cozzi:

Yeah, that's a great thought.

We wanted to switch gears a little bit and talk about Inworld and AI.

Earlier this year, you joined Inworld AI as Chief Creative Officer. For our audience that may not know, could you tell us a bit about Inworld?

John Gaeta:

Inworld was founded by some folks that were core engineers who produced work for Google Deep Mind and for a thing called Google Dialogue Flow, which is a conversational application that is... Well, it's many, many applications now used in Google products.

Essentially, formerly elite Google AI inventors and engineers that they're young people, they like playing games, and they like movies and stories just like the rest of us. So, at some point, they decided that it would be more rewarding for them to aim at a more focused use case.

What they wanted to focus on was characters. Very smartly, it's a very simple premise that they are putting forward.

What they believe is that games are full of these non-playable characters that are support characters that help provoke, prompt, or guide, or just stand around in your game to fill in the world and to help push you along in your experience.

They realize that there can be so much more to be done with that element in game design. Now their mission really is to basically build the minds of NPCs, and it will go a lot further than just support the game mission down the line, but that's the basic starter story.

Marc Petit:

So, John, what can we expect?

John Gaeta:

We've spent the last year speaking to some of the most remarkable creators, writers, IP makers, game designers, people who understand how to world-build and story-build.

We've been trying to understand, call it the mechanical techniques that they use, the devices they use to create depth in a character, not a one-dimensional character, but a fully dimensional character that has an understanding of self, history, and to some degree the ability to evolve over time based on change, context, inflection points.

We've been learning about what utilities, just to use a dry word, a writer might want, what devices a writer may want, or a game designer may want such that the character is not just a one-way information source.

What you should be looking ahead at is far more sophisticated uses and abilities of these characters to essentially progress across time with the player. As the player goes through the journey of their game, or otherwise missions and adventures, the characters can essentially learn and be updated as they go, as the world changes, let's say even, right?

There are different ways to look at it. Let's say you have a game world you want to populate full of characters; these NPCs have different roles. They fit archetypes. It's easy to hey, in Star Wars, you have droids, and you have rebels, and you have siths.

You can just divide up the character types into these archetypes, but of course, each one is an individual; they need a personality, a persona, and a history. They have relationships to other characters. So, you can start to create a structure, if you will, where you're building on top of archetypes as a base, and you can start designing really interesting surface personas and then start working on relationships of some others.

This is really fun. I mean, it's the first time that you could really see people who are tours of level design and mechanics making, working with a writer. It's a new intersection of talent that hasn't really existed quite yet up to this time. There's a lot of experimentation. It is no small task. It's going to take again a while of experimenting, optimizing, tuning, seeing how players and fans really decide to engage, and then learning from that.

It's really the beginning of a road. There really are things you can do today. There's a near, mid, far of this.

There are things you can do right now. You can already have completely unexpected kinds of conversations that make sense now that are not subject to hallucination and jumping the rails and all of that stuff. We're in the beginning of structured... They're not intelligent. They're trained, if you will. It's like the art of mimicry, right? Trying to mimic characters that we've seen before, but with new attributes.

Marc Petit:

That's fascinating. We could talk about this forever, but I think we're running out of time. So, Patrick, you take us home.

Patrick Cozzi:

Yeah, John, we love to wrap up by asking if you could give a shout-out to a person, people, or organization.

John Gaeta:

I will give a shout-out to all of the people who spend their many, many, many precious hours believing in a better, newer thing. They sacrifice quite a lot. Not everyone gets rewarded as they deserve, but without that community, things would just be really not fun.

It's just a general shout-out to all the innovators, the ambitious, the artists, and all the people I've had the luck to meet.

Marc Petit:

Well, thank you, John. John Gaeta, Academy Award winner, serial innovator, entrepreneur. You've been in so many interesting adventures, always passionate about immersive storytelling.

It was a pleasure talking to you today. We do hope the metaverse will open a new canvas for people like you to tell new stories. We're very much looking forward to that.

John Gaeta:

Well, thank you for the opportunity to riff with you.

Marc Petit:

Always a pleasure. Thank you, Patrick. Big thank you to our audience.

Feedback@buildingtheopenmetaverse.org is our email address. Let us know what you think. Let us know whom you want to hear from, and we'll be back for another episode. Thank you very much.