Blending Art and Tech: How Pixar CTO Steve May Pioneers Animation Tools and Aims to Standardize 3D for the Open Metaverse

Announcer: 

Today, on Building the Open Metaverse

‍

Steve May:

It's not about escaping the artist; it's about how you enable them to do more, be more creative, and provide them tools so that as they are working, you're just helping accelerate their creative process.

‍

Announcer:

Welcome to Building the Open Metaverse, where technology experts discuss how the community is building the open metaverse together. Hosted by Patrick Cozzi and Marc Petit.

‍

Marc Petit:

Hello everyone, welcome back metaverse builders, dreamers, and pioneers. I am Marc Petit, and this is my co-host, Patrick Cozzi.

Patrick Cozzi:

Hey Marc. I'm doing great. We're in for a treat today.

‍

Marc Petit:

Absolutely.

You're listening to Building the Open Metaverse Season 5. This podcast is your portal into the open virtual world and spatial computing. We bring you the people and the projects that are on the front lines of building the immersive internet of the future.

Patrick Cozzi:

Today, we have a special guest joining us on that mission, leading technology innovation at one of animation's most acclaimed studios.

Steve May has pioneered visual effects and production tools behind Pixar Classics as Chief Technology Officer; he now spearheads the development of cutting-edge pipelines with USD or Universal Scene Description and is leading the transformation of USD into an international standard as chair of the newly created alliance for AOUSD.

Marc Petit:

So, Steve, we'd like to hear from you, so can you please tell us your journey to the metaverse in your own words?

Steve May:

Yeah, sure. First, thanks for having me. It's an honor to be here. I'm not sure if I'm at the metaverse yet. I feel I can see it. It's on the horizon. It's coming.

My history is really about combined passion, love, for both computer science, technology, and art. On the flip side, I always wanted to do both, and when I was a kid, I kind of became aware of computer graphics as it was evolving, really at some very fundamental places like Lucasfilm.

The computer division of Lucasfilm was doing really cutting-edge computer graphics. That group would later become Pixar, and it was ironic that the image I saw was a very famous image in computer graphics from 1984 that was made by Tom Porter, and it was really one of the first examples of a rendering that had things like motion blur and soft shadows and really global illumination kinds of things in rendering. I saw that image as a teenager, and it just flipped the switch for me and decided that's what I wanted to do was computer graphics.

There was an article in a magazine called Science Magazine. It talked about the computer division and also talked about where I went to school, which was Ohio State University. So I read that magazine article, closed it up, and told my parents, "That's what I'm going to do. I'm going to go study computer graphics at Ohio State and maybe work for that Lucasfilm Company." It's especially ironic for me that when I did graduate from school and applied to Pixar, the person who interviewed me was the person who made the 1984 image in the first place.

‍

Steve May:

Tom Porter, who's one of the founders of Pixar and a pioneer in computer graphics. Up until the time that he retired, actually in the spring, our offices were immediately next door to each other after all of these years.

Marc Petit:

And it's been 25 years now?

Steve May:

I probably wouldn't have predicted that, but yes, I had my 25-year anniversary of Pixar this summer, actually, and my career has almost split in half. The first half was really working in production as one of the creative supervisors on movies like Monsters Inc., Finding Nemo, and Cars and Up and then the second half, really the last half as CTO of Pixar.

Marc Petit:

How has the pace of technological change differed between the early days? It felt like there was something new every day in the '90s and now. Do you see a difference?

Steve May:

Everything was still being invented, and we were still just trying to figure out how you make animation at all. How do you make 90 minutes of an animated film? That was still a gigantic problem when I started, and there were fundamentally unsolved problems. It was just constantly an awesome struggle and challenge to figure out, "How do we animate fur." I worked on Sullivan's fur technology for Monsters Inc. and making Sullivan. We had just never animated fur before and, in fact, really hadn't been done anywhere before that.

We had to write the software to do the grooming for the fur and the dynamic simulation for the fur. We didn't know how to shade fur to make it look like hair, so we had to figure that out. We didn't have good ways to calculate shadows for hair, so we invented deep shadows, which then led to things like deep compositing, but we didn't even have a good way to create shadows.

We were really figuring out just fundamental stuff. It was kind of the same. The next film I worked on was Finding Nemo. We really didn't know how to animate water. We just used whatever technique we could. Simulations took days and days to run, and they were not very good. I still have to cover my eyes when I watch certain scenes in Nemo because it's so crude and simplistic compared to what we can do today. It's remarkable.

Water is still hard and still expensive in visual effects and animation today, but what can be done is mind-blowing compared to what we did back on Finding Nemo. 

And one of the other things that's really interesting is the composition of the people making the movies here. When I started most of the artists here were generally referred to as our technical directors; most of them had to do some amount of programming to get their job done. Some of them did a lot of programming to do the job, even to assign material to a model, that required writing code back when I started at Pixar.

Another thing that's really changed a lot is that we've evolved from where most of the artists at least did some programming or were just programmers and writing the tools as we went. Today that's the exception. Most of our artists are really excellent, amazing visual artists. They're obviously good at using computers, but they're not writing code. The software is much, much more sophisticated.

The kinds of technology we do today are much bigger. Things like Presto, which is our animation tool, or USD, and RenderMan continues to evolve, but they're big, significant, very highly engineered software projects, and the expectations of the users is that those products and those things work very, very well. 

They have very high expectations for very streamlined workflows and very, very solid, reliable software. Whereas 25 years ago, the expectation was that it probably wasn't going to work, it would crash all the time and you probably would have to write some code yourself in order to get your job done.

It's changed quite a bit, and I would say the level of innovation in some ways is just as high, but it's more at these large impact big software project levels.

Marc Petit:

When you look back, is there a moment or a milestone that you look back on as being transformative or a turning point?

Steve May:

It's hard to single out single things. Compositing... I mentioned Tom Porter, who I worked with on Monsters Inc. and I interviewed with at Pixar. He invented compositing. That is such a ubiquitous technology that every person on the planet uses today because if you use a phone, it is compositing graphics layers on top of each other. There were just fundamental things like that, like developing RenderMan so that you had depth of field motion blur, things that you would expect to see in a live-action film so that you could then integrate graphics into live-action films in a meaningful way.

More recently, there have been just so many things when I started subdivision surfaces were really becoming something we used a Pixar, and that was a very fundamental technology. Path tracing, really in the last 15 years or so, fundamentally changed the way that we work. It fundamentally changed the scale of our data centers. Our render farms are much, much bigger than they used to be, largely to support path tracing, but that has enabled us to do things that we couldn't do before, too.

We really run dailies. We can now render the entire film or entire chunks of the film overnight or over the weekend and review shots in complete context so that when you're looking at even in-progress animation or in-progress effects animation or camera work or other things, you can see it in full context with lighting and everything else in the scene. We didn't have it before patch tracing and really big render farms. That really kind of changes the way you work because the artists are always looking at things in context.

There's been just a number of fundamental changes. There’s also things like the GPU that have changed the way that we work in terms of interactive tools, but also the way we render things.

RenderMan's gone through multiple evolutions. RenderMan's been around since 1988 as our rendering tool, and it's now in its third major iteration, which now really leverages the GPU to do things. There's just been so many things.

Marc Petit:

In Elemental, you pioneered volumetric rendering, too.

Steve May:

I get the question of like, "Well, what's still left to figure out and do? What's new in computer animation, computer graphics?" I'm like, "Oh, we're just getting started. We're still scratching the surface. We’re still struggling with really hard problems." Elemental was one of the hardest, technically, in some ways artistically, one of the hardest movies we've ever made because each character is a true volumetric render and simulation.

In Elemental, we have characters that are made out of fire and water, air and earth, and in particular, the water, air, and fire characters. These are volumetric simulations that are computed dynamically on every frame and then rendered as volumes and the amount of storage you need to store all that volumetric data. You can imagine a scene where we've got a crowd of characters that are volumes, and then the computation required to render those volumes is extreme.

We had to really rebuild our data center to support making that film. Artistically, it was super challenging, too.

How do you read the facial expressions of a character that is an evolving volumetric fire, but you need to have appeal? The character needs to be appealing. You need to be able to see the eyes and the mouth and the mouth shapes, and you need to be able to animate very expressive poses to connect with the audience. That's an extreme artistic challenge.

I love that. Even today, after all these years, we have films like Elemental that really push the envelope as far as what's possible and what we think we can do.

Patrick Cozzi:

When you look at making a film today, how are you balancing supporting the production needs vs those longer-term R&D initiatives?

Steve May:

It's a challenge. It's a challenge. One of the things I would say that is unique and awesome about being a technologist or software developer at Pixar is that you make really state-of-the-art software for a very small number, like a few hundred, of the best animators in the world; you're not making software for thousands or millions or billions of people. You're making it for a few hundred. They're the very best at what they do.

By the way, their office is right next door. You'll hear about it if the software is not great. But also that ability to sit next to an animator and learn how they do what they do and then in real-time work with them to solve problems and evolve the technology is super special.

The downside is that that amazing connection makes it hard to carve out projects that are not being driven by the active productions, the active films that we're trying to create.

We do balance it. We do balance it. We're a movie studio. Our job is to make the film. Mostly we’ll drive things based on what the movies need, but then... For example, we mentioned USD, Universal Scene Description; that is a project we carved out because we identified that as a long-term problem we wanted to solve within the studio.

The movies were not asking for that at all. I had to cajole the first show to use it, to kind of take it because that was not driven by them, but it was a priority for us at a studio level. And so I guess the answer is we just try to build that into our project planning, and it's really hard, but the fact that it's hard makes it so much fun to develop technology at a movie studio.

Marc Petit:

How has your technical team evolved over the past years? Do you recruit differently? What's the kind of profile that you're after?

Steve May:

There was a time when I probably could have been a developer in our software teams. I did have a software engineering education and did write a fair amount of software, but the level of sophistication of our software engineers is so much higher than it used to be.

Writing a GPU-based path tracer in today's world for major animated feature films or visual effects production like RenderMan requires a very high level of skill, knowledge of computer graphics, and software engineering ability. We see that not just obviously in RenderMan, but in our animation system in Presto, in USD.

I'm blown away by the skill level of the software engineers that we have. I think that's the biggest change. Otherwise, I think, in many ways, things are similar, especially in the way that we try to make sure that engineering and animation production are really close with each other.

Patrick Cozzi:

Steve, I wanted to ask you a bit about the leadership and culture side of the house.

As you know, last season, we had Pixar co-founder Ed Catmull on the podcast, and we love his book Creativity Inc. Actually, at Cesium, every new person who joins, we give them a copy of Creativity Inc. We say, "Hey, this is how we aspire to act."

Could you tell us about leadership and culture nowadays at Pixar?

Steve May:

We refer to that book often here still, and Ed still stops by on a fairly regular basis, and if he does, there's a line of people wanting to ask him questions and get his input. The thing about the culture of Pixar, Pixar is over 40 years old as an entity; the culture we have today is the accumulation and result of many, many years and decades of the things that people like Ed brought.

Things always change and evolve. All companies are living organisms. They have to modify the way that we do things. But when it comes to a lot of the things that Ed was really formational about here, we carry that on not only in his honor because it's really the smart thing to do.

Ed's background in academia, and I have some of that too, we really put an emphasis on research. We put an emphasis on publishing papers, presenting our research, sharing technology through open source projects, all those things when it makes sense because we believe that if we have that sharing... I would say one of the great things about our community and in general in the film industry, visual effects, computer graphics, and animation is that we are a collaborative community.

Even though it's a competitive space, we're collaborative. We realize that if we share things, especially things that are kind of baseline technologies that we can all benefit from each other's developments. We feel like if we publish a paper and then ten students read that paper and then they write ten papers and then that will come back to us and benefit the studio.

I think that you can see that in many different ways, and I think that's still a fundamental principle that we go by.

Marc Petit:

You and I were on a panel at SIGGRAPH talking about AI. So we have to touch on the sujet du jour.

Are there any specific capabilities that get you really excited to deploy in production over the next few years?

Steve May:

What I think is going to be exciting is the prop-based stuff is great, and it's amazing, and there is an important place for that as we go forward. But I think the way that we'll really manifest it is... We call it Artist in the Loop, which is... It's not about escaping the artist; it's about how you enable them to do more, be more creative, and provide them tools so that as they are working, you're just helping accelerate their creative process.

It's not about doing something where we can just bypass the creative process or bypass that talented artist we have.

The other thing is that we are very, very, very specific about how we animate things. The shape of the profile of a character silhouette is something that we fine-tune within, like... We would call a pencil's width; that kind of adjustment. I think where we are not at with most of the AI techniques is in that level of direct ability or control. That's what we would have to have to use AI to do most of the things we do.

We still do many things that are very tedious and time-consuming, and probably are not that much fun for the artist. If we can create those tools where the artist is in the loop, and you're just helping them accelerate and skip over the parts that are more tedious, provide more inference and suggestions for what they might want to do next rather than just having them manually specify every single thing, I think that's where there's a ton of potential.

One of the benefits of Pixar is we've got this great culture of innovation, but we also work within this bigger company, Disney. We very closely collaborate and share technology and learnings with the other studios within Disney, like ILM and Disney Animation, and Marvel, but also Disney research.

There's a lot of interesting experiments, but one example is when we're posing a character, what we generally do is we have thousands of controls for every character, literally thousands and thousands of controls for each animated character. Most of those controls are hierarchical, so they're higher level controls like a pucker control for your mouth, for your lips on an animated character to make an “ooo” sound. But a lot of times, it's individual controls.

One of the things that we're looking at is based on our existing set of poses that we know about in our training, of all the data that we've got here because we've got the data from all of our films all the way back to Toy Story. It's still online and accessible all the time here at Pixar. Can we do a better idea or a better job of guessing that if someone moves an elbow joint, would the hand likely follow it instead of them having to move the elbow and then move the hand?

Marc Petit:

So it's about augmentation and acceleration?

Steve May:

It really is.

I haven't seen anything yet where I'm like, "Oh wow, you could do that instead of having an artist do it." I don't see that happening anytime soon, nor do I really want that kind of world where that happens. The creativity comes from the spontaneity of the artists as they work. We often don't know what we want until we start working on it. In the process of working on it and making it, then we realize what we're doing, and that's what creativity is.

Patrick Cozzi:

Steve, you also mentioned Disney. Pixar recently announced it's working on its first long-form animated series for Disney Plus. I was curious if you could share a bit about how your approach to the series would differ compared to feature films.

Steve May:

The pipeline is essentially the same. We're not changing software.

We still use Presto, and USD, and RenderMan, and on our other tools and also commercial tools like Houdini, and Katana, and Maya. The thought process is the difference; with the series, we've got more footage and more minutes of animation to produce. The economics of streaming series are such that we are trying to do that with basically a smaller budget per minute.

Really, the main thing is being creative about how we think about solving the storytelling problems in streaming and how we alter our mindset about features because we demand the same quality of those streaming series that we demand of one of our feature films.

What's been really inspiring and great about it is that shakeup of, "Hey, you need to make more footage for less and get the same kind of quality" is that it really makes you rethink the way you do things. It's been more about how we modify our process, how we run our creative reviews. It's been freeing to say, "Hey, I don't want you to do things the way you do them for a feature film. Think about different ways to do it for the streaming series."

One of the great benefits is that those new learnings and thoughts can now flow back into how we make feature animated films so that we can make our feature animated films even better, too.

Marc Petit:

I also remember you talking about new mediums like AR and VR and mentioning that... Just to figure out what immersive storytelling really means.

What have you learned so far with those new mediums?

Steve May:

Our job is still to make movies and linear streaming content, but we have done experiments with immersive kinds of storytelling. I would say what we still know today is that it's really hard to make really, really compelling stories in a way that can connect with the audience at an emotional level. That's not to say that it's not possible. I think that will happen; it's just still a new medium. We're still figuring it out.

It took us a long time to figure out how to do that with movies, too. And the potential, I think, especially for Pixar and for Disney, is really compelling.

Fans of our movies and fans of Disney content, in general, are really big fans and really want to continue... After they've watched a movie, they want to continue to connect with the stories and connect with the worlds, and connect with the characters. You see this in the devotion to things like theme parks, things like consumer products, and even Disney theatrical, the stage plays, musicals, that Disney does.

It feels like it is such an opportunity for a company like Disney and studios like Pixar to continue to engage with the audience after they've seen the movie, and in a completely different way.

I don't know any company that could do that better because we already have immersive things. They're called the theme parks. Disney really knows how to do this. We have these great stories, we have this great legacy of characters and worlds, and it just seems like the obvious thing is the next step is to bring in interactive 3D immersive content and connect all those things together in a way that I think would be really fun for audiences and would make a lot of sense for the company. But it’s early days, and we're figuring that out, but I think the potential is super exciting.

Marc Petit:

I think I saw Bob Iger on stage at the launch of the Apple Vision Pro. Everybody did. I was actually the only person outside of Apple present at that announcement.

Do you think that device, or maybe not the first one, but the kind of device, has the potential to be transformative?

Steve May:

That device is very impressive technologically, and Apple has a track record of making platforms that gain wide adoption by audiences. I think all of us that are interested in immersive content are optimistic and hopeful that we're moving along this path to where we can really have that kind of content be more available to a broader audience.

We're talking about the metaverse in this podcast, and I think the metaverse is really about getting 3D content, either creation or consumption, beyond the walls of computer graphics experts and people to do effects and animation and gaming and into, I would say, regular people's hands in air quotes like our friends, our family, people who don't nerd out on the same stuff that we do.

That's what it's about. It feels like we're on that path, and the things that were shown in that keynote were very exciting both from Apple, but also the ideas of the things that were shown in the part with Disney, I think, give a window to what that looks like.

Patrick Cozzi:

Steve, I wanted to ask you about real-time in general, right? So, you talked about all the advancements to production and GPU path tracing. How do you see a real-time 3D in a complex pipeline like Pixar's?

Steve May:

Two sides of this. In one way, we're like the opposite of real-time. We still make movies, and we make streaming content, and that's basically a recorded format.

Even though it plays back at 24 frames per second or 30 frames per second, we could take as long as we want to make each one of those frames because it's recorded. Our renders today... It sounds ridiculous, but a single frame of one of our films can take tens of hours, sometimes hundreds of hours per frame, to compute the final quality.

There were frames in Elemental that took hundreds of hours per frame to compute. On one hand, we're like the opposite of real-time. On the other hand, that's how we want our artists to work. You don't want to wait for a result; you want everything to be interactive.

Most of our pipeline has gotten very interactive, or you could call it real-time.

There are still aspects of art in general rendering, though final quality images are clearly not interactive, although we're working on that really hard with RenderMan's view and making some big strides there. 

Things like lighting are still not real-time. Some forms of simulation, like complex cloth simulation, are not real-time; we worked really hard to make advances in innovation so that the feedback to the artist is as interactive and real-time as possible.

Again, it's just another way to enable more creativity and make a better creative process.

It's hard to even explain to some people who haven't used simulation software or complex lighting software to understand that you have to wait to see the results of your change. I've equated it to, imagine you're playing a guitar and you strum a chord, but you don't hear what it sounds like for a couple of minutes, and then you have to say, "Oh, okay, that's not what I want. I want to change it." And you strum another chord and I have to wait a couple of minutes to see the results again. Clearly that's not ideal.

We're working really hard. We talked a bit about AI besides generative AI that can produce some kinds of content. Machine learning, in general, has opened up many things for making much faster renderers and much faster simulations, character rigs, and other things like that.

We're seeing benefits from that already in significant ways that give us much faster real-time feedback for our artists and animators.

Marc Petit:

Well, let's switch gears a little bit and talk about another topic.

Pixar has a very long tradition of contributing and even raising open source software, OpenSubdiv, USD, of course. Did you initiate OpenTimelineIO as well?

Steve May:

Technically, you could go back to the RenderMan interface specification, which was published in the 1980s. That was really an open... Did not necessarily succeed as a very broadly adopted standard, but it was an openly published standard.

In modern times, OpenSubdiv was the first, and OpenTimelineIO is our most recent open-source project.

Marc Petit:

How do you determine what needs to remain proprietary versus what gains from being shared?

Steve May:

We have a variety. We have a range of technologies here, and RenderMan is a commercial product, USD and OpenSubdiv, and OpenTimelineIO are open-source projects.

There are even some things that we might keep as just proprietary trade secrets, et cetera. The approach that I take is to look at each technology and think, "What is the best way to use this for the most benefit to the company?" That's what determines how we apply it. In some cases, the best route with the technology is to publish a paper on it because we think that'll give us the most benefit back. 

As I mentioned earlier, students read that paper and do research, which can come back to us as a benefit. In other cases, it is to keep something proprietary, and in other cases, the benefit is to open source, and it's really just based on what will give us the most strategic value.

The subdivision surface technology is an interesting case study because, again, that was our first one. We created the subdivs for sure. We created that technology. Actually, Ed pioneered that with Jim Clark originally in the 1970s, but it wasn't until the 1990s that we came up with a more practical formulation of it and papers were published.

Tony DeRose published an excellent paper on it. But we found that even with that paper and all the detail that was in it, it was hard for others to implement it exactly correctly and to match the way that we used it internally at Pixar. So it would be great because it would be adopted, and someone had incorporated it into some software that we would use, but it'd be slightly different than our answer internally or what RenderMan would produce if you rendered the same mesh and we realized that... We weren't really getting the full benefit of the technology.

We wanted all the other software we buy, that's commercially availabl,e to do it exactly how we want to do it, so we would match our internal software.

The best way to do that was to provide a reference implementation. If we made an open-source version that was available to everyone, they wouldn't have to guess about the math of the implementation details. They could just take that reference implementation and incorporate it into their software, and then it would work the way we wanted it to work; everything would match between different software applications, whether they were external ones or internal ones, and that was where we would get the most benefit from the technology. That's why we open-sourced it.

Marc Petit:

Before we talk about USD in the spectrum of open source versus commercial, like RenderMan, where do you think Presto belongs?

Steve May:

Okay. First of all, a lot of Presto is in USD.

Especially as we continue to expand the capabilities of USD, effectively, more and more of Presto is basically open source because it is in USD. I wouldn't say it can happen. It won't happen. The question does come up about Presto and even things like RenderMan, but there are no plans to open-source either one.

Patrick Cozzi:

Earlier, you mentioned that when you think of the metaverse, it's going to enable everyone to do 3D creation and 3D consumption.

I guess if you'd look forward five or ten years, is there anything else you'd want to add to that creative metaverse vision and then how you think USD and other technologies could enable this?

Steve May:

Obviously, we're big believers in USD and I do think that could become the common currency of the metaverse. That is the way that we'll exchange 3D content as we go forward.

Again, I think it's such a broad potential set of applications that we talk about. If we talk about 3D content in the future, it's a little bit hard to put a pin in the bigger concept and impact.

I think from a Pixar standpoint, it’s more about the potential for storytelling and connecting with audiences, and that really means making it something that is accessible. Things like USD and open standards are key to that, accessible to a wide audience. By making it accessible to a wider audience, I think we'll also include in that is by making it accessible to a wider audience of creators.

I think whether those creators are within Disney, or Pixar, or elsewhere, we'll figure out really exciting ways to use immersive 3D content as a storytelling medium. We'll start to really figure it out.

Marc Petit:

USD is being used for real-time and many interactive applications. Did you ever expect that? You've done a good job telling us you guys do movies, you're focused on making movies, and now we see USD at the heart of real-time platforms.

What does it mean for you guys?

Steve May:

We expected USD would be successful within the film industry. We feel pretty confident about that, including for... I mentioned before even that our tools... We have a lot of interactive tools to leverage USD in-house. Even our animation system can run in real-time if you simplify the scene enough.

But I really did not anticipate it going beyond the film industry. That has been a really pleasant, exciting surprise to see the interest in adoption in industrial applications, in areas that... I think Marc, we did a podcast, or sorry, a panel last year where we had people from completely different companies and industries, including car manufacturers and home retail hardware stores, and it's like, "Wow, we did not anticipate that.” And that's super exciting. I think there's some fear that could fragment or dilute our use of it for film. I think that's a reasonable thing to be worried about. On the optimistic view, I look at how all those other interesting worlds and applications and industries could give very interesting technology back to us.

Computer graphics has a long, long treasured history of adopting science and technology from other fields and incorporating them into our computer animation.

For example, dynamics and simulation are prime examples, but also just rendering and pulling from fields like optics and other areas. I see that as just an enormous opportunity that other companies will think of ways to use USD that we never envisioned and that will benefit us. It feels like a real honor and validation of a technology when it gets used in ways that you didn't anticipate.

That's why it's so satisfying to see it being used in ways that we never really would've guessed.

Patrick:

One of the magical parts of USD is enabling that collaboration at scale. Could you tell us a bit about how it works?

Steve May:

There are multiple, I guess, axes of how it enables collaboration.

One aspect is the great focus by the engineering team on performance, and making sure, for example, that you can load, manipulate, and edit very complex scenes very, very quickly and interactively.

But I would say the biggest aspect is the way that it basically allows you to aggregate and assemble scenes from a collection of individual USD assets. At Pixar, we have many people working on a given shot in a given scene at the same time. USD allows you to compartmentalize the work into layers. In the same way that when you edit one layer in an image editing software like Photoshop, you're not breaking the other layers. That's how we think about animation and building our films here; we have artists working in different USD layers and they can do their work, but at the same time, you can see what the complete scene looks like by compositing all those layers together and seeing them in context.

That's the key for collaboration.

USD is based on the technology that we've been using to make movies for many decades, preceding the creation of USD. That was all about how you let a lot of artists work on the same content together and build very, very complex worlds and 3D scenes. I think the most fundamental part of that is the ability to layer and then also reference or pull in other models to make one aggregate scene from a bunch of individual parts. It's the ultimate divide and conquer.

If you're making a very, very complex thing, you don't have to think about it as one entity. You can break it up into individual small pieces, and those small pieces can all be edited and manipulated and created by individual artists and then combined together in a very efficient way.

Marc Petit:

You are the chairman and co-founder of the Alliance for OpenUSD, which was created a few weeks ago.

Tell us about the charter of the organization and your goals.

Steve May:

Well, the goal is to evolve. USD is on this path.

We've talked about the impact; it could become the currency of 3D content going forward. The next step on the path, the open source project we opened sourced in 2016, has been available for quite a while. It's been in use at Pixar since Finding Dory, the first movie we used on it here.

Along this path of developing it, using it, open-sourcing it, and getting industry adoption from it, the next step is really to try to make it into an international standard, a formal international standard. That's really the core goal of the alliance, which is to evolve USD into an international standard so it can become that currency we all hope that it will be and that we need for interchanging 3D data in the future.

It's pretty simple. That's what it's about.

Marc Petit:

Have you guys started to work on that?

Steve May:

The core working group, which specifically is creating the specification, is just getting up and running. So, exciting things to come.

Patrick Cozzi:

And Steve, are there any just parts of the roadmap or ways that you're hoping that new partners will help contribute and evolve USD?

Steve May:

Applications in architecture, construction, and industrial applications are ones that are very interesting and not Pixar's expertise. I'm looking forward to just seeing the things that come from those other areas. 

That would also include... I should also add interactive components that you would be interested in for immersive stuff or games. Internally, though, even at Pixar, we sell a lot of things that we want to contribute back into USD, including animation curves, things like rigging for characters, which, by the way, also requires having some sort of execution system.

We're very busy right now on both the animation curves and the execution system to lay the foundation for the next generation of the kinds of content you can make in USD, in particular as far as animated characters, avatars, and that kind of content.

Patrick Cozzi:

Steve, I wanted to ask you one last topic.

You were talking about all the papers that Pixar's publishing and how it's great that students get a hold of them and do additional work also, I'm sure an amazing recruiting tool.

As you know, I used to teach in the graphics program at the University of Pennsylvania, and many of our students have gone to Pixar. I'm doing everything I can in the Philadelphia area to help build a graphics presence with all the other universities.

Do you have any asks for things that you think you'd like to see in graphics programs at the university level?

Steve May:

Well, first of all, thank you. Penn's been a great source of talent for Pixar over the years. I think the main thing, the thing that I always–if I talk to people at different universities or colleges, is finding ways to interject art and technology together.

If you're a computer science professor, find ways to connect with the College of the Arts at the university or to bring in students from other areas, not even traditional art. It could be music, it could be drama, it could be other forms of art.

I still feel like that's where the magic, I don't know, sauce is. Steve Jobs, the founder and original CEO of Pixar, was big on that intersection of technology and liberal arts. That's where there's just an enormous amount of potential there as we go forward, especially as technology becomes more sophisticated and more advanced; it feels like it's increasing at an ever greater rate.

Having the sensibilities of liberal arts and artists involved is really important to us as a society. But selfishly, as a person who works at Pixar, computer science students, computer graphics students who have sensibilities around art, that's the secret thing we're always looking for.

Marc Petit:

Thank you, Steve. This will take us to the last question of this podcast, which is our traditional ask.

If you have a shout-out that you want to make today to a person, an organization, an institution, or a company.

Steve May:

I'm going to keep it simple, I'm just going to give a shout-out to the software engineers we have at Pixar, both in our two main groups, Tools and RenderMan. They are the best. They're world-class. They work extremely hard.

Since we're talking about the things we've been talking about with USD, the USD team is amazing. We've been growing that team a bit to handle all the new things we want to do with USD, and we've got great leadership there.

Sebastian Grassia, we all call Spiff and Florian Zitzelsberger there, our two leads of that project, they're doing an amazing job. And so I just give a shout-out to everyone at Tools to develop software and works with our software teams.

Marc Petit:

That would add a shout-out to Nick Porcino; Patrick and I are at the Metaverse Standards Forum and he has been such a positive force to bring people together and to drive that interoperability. And honestly, we owe him a lot in terms of the progress we're making aligning glTF and USD. Nick Porcino is a rockstar, so thank you for letting him spend time with us.

Steve May:

I do want to say that in addition to Nick, we have this amazing engineering team in general with USD that does all the behind-the-scenes work.

Huge credit to all of them for the work they do. They're the ones that actually make this happen.

Marc Petit:

Well, Steve, it was an honor to learn about your pioneer work at Pixar, fusing technology and storytelling.

We're truly inspired by your approach, crafting custom tools, commercial tools, open-source projects to enable the magic of filmmaking. Your passion for blending engineering and creativity is really truly remarkable.

It's remarkable to see you lead the creation of a standard that might be the foundation of the immersive and special internet of the future. That open metaverse that we all talk about. Fantastic job. Thank you very much for being with us today, and Godspeed.

Steve May:

Thanks for having me. It was a pleasure.

Marc Petit:

And a big thank you to our listeners.

We did 56 episodes over the first four seasons. We're into Season 5, and we're super happy to get the feedback.

You can follow us on LinkedIn. We have a page there where you can find all those episodes. And on our webpage buildingtheopenmetaverse.org.

‍