Cloud Rendering on the Blockchain

Announcer:

Today on Building the Open Metaverse.

Jules Urbach:

Streaming from the cloud is just a GPU cost. Basically, if you're building up an app, a GPU on Amazon to stream anything, you're still paying for those GPU minutes. If it's an offer on Render, you're paying for those GPU minutes in OctaneBench because it's a high latency job. But imagine that you could do the job in real-time at 60 frames a second with just more GPUs. Like Octane scales, we get 100 or 1,000, we showed this in 2013 when Jensen brought me on stage, we had 100 GPUs and it practically was real-time. Now, it only takes five GPUs because this was 2013. But the idea is that really, the idea of Render and the OctaneBench score or the compute cost of Render is something that you could just add more power to get it to be in real-time or simplify certain steps of the rendering process, but you're still paying for GPU cost.

Announcer:

Welcome to Building the Open Metaverse where technology experts discuss how the community is building the Open Metaverse together, hosted by Patrick Cozzi from Cesium and Marc Petit from Epic Games.

Marc Petit:

Hello, everybody and welcome to our show, Building the Open Metaverse, the podcast where technologists share their insight on how the community is building the metaverse together. I'm here today, I'm Marc Petit. I'm with Patrick Cozzi, my partner in crime for this podcast. Hello, Patrick.

Patrick Cozzi:

Hey, Marc. Hey, everybody.

Marc Petit:

And today we have an intriguing guest, somebody who's been in the industry for a long time, has been funding one of the important CG companies in the industry. Our guest today is Jules Urbach, CEO and founder of OTOY and as well as CEO founder of RNDR, which we're going to discuss and you're going to explain to us what the RNDR is. And so, Jules, welcome to the show, we're super happy. We know you've been talking about the metaverse for a long time, so we were very, very looking forward to have you on the podcast.

Jules Urbach:

It's a pleasure being here, Marc and Patrick, it's a pleasure, yeah. It's great, thank you.

Marc Petit:

So maybe start by introducing yourself and describe your journey to the metaverse and why you're with us today?

Jules Urbach:

Absolutely. Well, my background, I've been in the business now since 1993, when I did my first CD-ROM game, Hell Cab. This was in the days when CD-ROM was a new thing. So going almost on 30 years in the industry and my passion has been both video games and films. And so, OTOY really came about, I mean, it was just me starting the company in my bedroom at my mom's house 20 years ago. But the vision of the company was to take some of the lessons I had learned from the video game world, my passion for CG and for films, and to kind of blend those together. And that's really how OTOY started, and that's something that was never really the destination either. It was all about building virtual worlds in a connected space, these mirror worlds that were kind of already in sort of the pop culture in terms of people's minds of what it could look like.

Jules Urbach:

And for me, things like the internet, and the web, and social networks, and video games, and films, to me, are all just precursors to something that combines all of those. You could imagine narrative elements that are holographically experienced, even if they're intended to be told like a story that's been in books and movies before. And games, for me, are something that, again, you're starting to see that the interactive elements from all of these different medium's kind of converge.

Jules Urbach:

And I think the first two decades of my work, starting and building OTOY, was about figuring out how to get the quality of real-time rendering practically to the point where we could have one product that could do both films and games. And I think that's something that's starting to really become possible with ray tracing hardware that's been introduced by first Nvidia and now others. Obviously, with Unreal Engine, I'm a user, we have tons of artists in our company that use it. And OTOY builds a software render or a GPU render called Octane, that's integrated in every 3D tool for production and high-end rendering, but all these things are really about just democratizing the tool set.

Jules Urbach:

And a lot of what OTOY is best known for is OctaneRender, mostly among motion graphics, artists that use it create beautiful graphics that are rendered on the GPU. And we disrupted a lot of the thinking around offline rendering back 10 years ago when we introduced Octane, because we were able to go to full unbiased spectral rendering on people's laptops at 40 times the speed of CPU rendering, and we still do that.

Jules Urbach:

But I think that the future of where this intersects with the concept of the metaverse is that I do feel like there's no point in trying to do less than photorealistic rendering because there's going to be at least some portion of the world that wants to see a simulated virtual world that looks just like the real one, including people that make movies and CG, that's always been the holy grail, is CG and anything you can throw in front of the camera and do in real-time is obviously good enough for films, but also something that games have strived for as well.

Jules Urbach:

And where I think games have become very, very interesting is in the rise, frankly, of social networks and the crazy amounts of investment since Oculus, I think, was acquired back in 2014 around the concept of building out experiences. And there's just been this draw line where we're not even really seeing the beginning stages of true real AR and VR adoption. I think that devices need to get smaller and more comfortable, but that might happen this year, next year even.

Jules Urbach:

And then on the other side of things, in the last 10 years, one of the things has caught my attention separately from all these other pieces has been the blockchain. I mean, I was tracking bitcoin back in 2010 and '11. I, previously to that, was thinking about the hard problems to solve for rendering a mirror world or the metaverse, whether it's real-time or otherwise, and compute cycles or something that's a resource like energy, like oil or electricity, literally it's power.

Jules Urbach:

And my concept back before GPUs were a thing and in the cloud was to take all this compute power from graphics cards that were out there and pay users per ray that was traced, even if it were a penny. That was where the genesis for the Render Network really came from, it goes back almost to the beginning of '02. I was thinking about this even in 2004, and that was my plan B. My plan A was maybe Amazon, or Microsoft, or Google will put GPUs in the cloud, and then cloud gaming, and cloud rendering can happen. That sort of happened, but the costs and the capacity are still limited. And we are one of the largest consumers of GPUs for clients that wanted do offline rendering and real-time streaming.

Jules Urbach:

So the Render Network was a project that kind of came into fruition or focus for me in the 2016, 2017 timeframe with the rise of Ethereum and the ability to sort of use the value of both cryptocurrency tokens. And the fact that GPUs were used to my cryptocurrency, I figured, "Okay, well, this is the perfect time to reintroduce this concept from 13 years ago where people can get paid," and now they're used to getting paid for renting their GPUs for some larger network.

Jules Urbach:

And what we did is we just took the services that we were running both the real-time streaming service, which had been running since 2014 or '15, streaming, Unreal, Unity, anything, any kind of app or game that was a service that we built initially for Autodesk, and also offline rendering, which we had done for our production rendering pipeline with Octane and we transposed those to the Render Network. And we took the cost of what it would take to render in terms of GPU time for ray on Amazon, Google and Microsoft, which are largely the same seven years later, and we pay out a cryptocurrency token as an ERC-20 token to those miners.

Jules Urbach:

And what's amazing is that even at a fraction pf the cost of what cloud rendering or computing is on AWS, we are able to actually have this huge wait list. I mean, we have like 100,000 nodes in the wait list and just enormous amounts of GPU power because as long as it's more expensive than electricity and pays up better than Ethereum mining or any other cryptocurrency mining, people want to make money. And a lot of, even our own customers have tons of extra GPU power that they've had on premise to do GPU rendering that they now are not even using. They're sending their render jobs to their Render Network.

Jules Urbach:

We have hundreds of GPUs per customer practically, and you get back things in some cases, it couldn't even be faster than real-time. If you were to imagine having enormous amounts of GPU power for every frame, a two-hour real-time project where it could still be divided into many thousands of frames that are done in parallel. So the parallelization of rendering is something that the Render Network is excellent at, and ultimately, I see that also just being a necessity for the size of certain assets or the ability to resolve virtual worlds or portals that just don't have the bandwidth or the ability to render every device in the planet. So that's kind of where the story of OTOY and Render has taken us.

Jules Urbach:

And then when it comes to the metaverse, that's a word that is very loaded. I love the idea of it. I mean, obviously canonically, it comes from Neal Stephenson, going back to what, '92, and everyone's sort of been thinking about having a world you can plug into and experience. I think that that is something that is absolutely fundamentally in the future world line of humanity. There's, of course, many thoughts philosophically around, are we living in a simulated reality? Not even necessarily computable one, but if the laws of physics themselves are so precise and yet certain constants, isn't this all sort of just potentially all code?

Jules Urbach:

And I do think that the metaverse just allows us to consider the digitization of almost everything that's physically out there that hasn't been digitized. We've gone through so many forms of physical mediums, and experiences, and things that have been transcribed and digitized. I mean, we have knowledge, we now have communication and all these things, it's really sort of the last frontier of digitizing the physical world is recreating the experience of being in the physical world and making that something that is a shared collective conscious experience. And of course, when you imagine that as a foundational baseline for many other types of services including work, and learning, and entertainment, and storytelling, and games, and possibly even research and exploration, that to me is the potential promise of the metaverse just in our creative, and artistic and humanist level.

Jules Urbach:

I do think that what's happened since Facebook changed its name to Meta is that, and we've seen this, there's a gold rush to just putting anything that you can, connected to this label. And I think that's a bit sad because that everything is the metaverse and not everything that says they're in the metaverse is really in it yet. We're still at the protocol level, in my mind, of what the metaverse needs to be, and so saying something's in the metaverse is a bit tricky. It's like saying... it's just a bit early.

Jules Urbach:

If I were to sort of gauge where things are at, I would say that before we had the internet and the web, we had DARPA, before there was the browser, you had TCP/IP and you had UDP, you had protocols that just transferred data, then you had things like the web, which was something that really did transform society and opened the door to the multi-trillion-dollar companies that we see today. Google, and Amazon, and Facebook, even Apple, I think owe a lot of their early growth to the fact that you they had Safari on the iPhone 1, there was no App Store. That was how you would see and experience things.

Jules Urbach:

And I lament the fact that since the applification or the App Store was launched, and since we've had, even in some ways, the walling off of the open web into social media verticals, we've lost a lot. And my hope is, and it's shared by one of my mentors and advisors to both OTOY and Render, Brendan Eich, who created JavaScript when he was in Netscape. He also created Mozilla, and Firefox, and went on to create the Brave browser as basic attention token. He was telling me that if you'd only put a buy button in Mozilla's Firefox, it might have totally changed the world. You might not have had the apps remodel evolved. The open web could have actually on something that would've been a platform.

Jules Urbach:

And they were working on the Mozilla phone. Andreas Gal, who's now at Apple was the CTO of Mozilla, was developing essentially what you have now in the browser where you can provide the camera and the microphone services for webpage per site, progressive web apps, and all those things combined would've been an amazing open frontier for development. And now you have a duopoly of App Stores, you have sort of this limited set of things you can do within those frameworks, and I hope that doesn't happen for the metaverse.

Jules Urbach:

And I know I've been sort of talking everyone's ear off to this point, but it's such a big part of sort of why I want open standards is I do feel like as a creator developer, I've always wanted my software, my art to go everywhere, to not worry about it being limited, to not have the ability of it to not be shared. And I think that I've seen a lot of... I mean, starting right, I mean, obviously everyone's aware of the fact that Microsoft in the '90s was a bad actor. I mean, Internet Explorer, was building active X controls to put games and browsers, and there really wasn't anything. I mean, it was whatever Microsoft said the web was, was what we got.

Jules Urbach:’

And the reason why I really respect Brendan is that when he went and did Firefox and Mozilla, and flipped that whole strip around, I mean, we all are using maybe Chrome or Safari now, but that was something that really did transform the world. And I think the metaverse needs that Mozilla model, that's what I hope the metaverse is, and I do think that that's where we want to take it. So on that note, I think that sort of describes a bit of my journey to the Open Metaverse.

Marc Petit:

Yeah. Thanks for going straight to the heart of the matter. I mean, that's good. How do you make the Mozilla model happen in the metaverse? You have some ideas, lectures, how do you make the Mozilla model a reality for the metaverse? Collectively in an industry, what do we need to do?

Jules Urbach:

Well, the first thing is don't let the equivalent of Microsoft hoarding off 51% of it, that would be that because that's the first thing, is I think that the danger is simply that somebody gets so much traction in the space that it doesn't even matter what anyone else does, and that's always a problem. So I think we're lucky in the sense that there's multiple trillion-dollar companies that want to play in the space that probably won't let other multi-trillion-dollar companies have that 51% ownership of it, but it doesn't really help either if you have a duopoly or something along those lines because there is... This is a much more complicated system than anything that preceded it.

Jules Urbach:

In some ways, if you look back at the Mozilla model, it's a 2D document model. I mean, the document model has been around since the Gutenberg printing press or before. I mean, this is something that... The metaverse is very different, it's spatial, obviously. I think if people just largely read that in a lot of respects. It's built on a foundation where space, and time, and spatial rendering, and all these things are just fundamentally a part of it as much as data is. And so, when you talk about building something in the metaverse and having that same thing operate in somebody else's part of the metaverse, then things like open standards and interoperability really do matter. And of course, if you have anything that's commercially locking in somebody into one app payment system or the rules for that payment system, I mean, that becomes complex.

Jules Urbach:

So one of the best [inaudible 00:16:13], I think, to a closed vertical centralized metaverse is decentralization, is crypto, and that's why I'm such a huge proponent of it. I think the payments alone are largely, in some ways, solved because if you look at cryptocurrency and there's going to be many, many, many different alt coins beyond Bitcoin and Ethereum, and the big ones even like Solana that we're partnering with them too, the idea of sort of paying in a format or in a way that provides value outside of even money now is a thing, that's amazing.

Jules Urbach:

And you have storage that's decentralized, so you have Arweave, you have IPFS, you have things like that where we're standing in the point where certain fundamental things don't have to be on a public cloud or on a website, and that's sort of where even Render comes in, if we can decentralize and democratize computing resources, storage on all those things, and we have payments kind of covered. Then I think at a foundational level, we have some of the things sorted out that were never sorted out in the App Store or web model at all. I mean, payments are still kind of a bit tricky, whereas when you go and you let's say buy an NFT, typically, you could do it with a crypto wallet. There's still probably a lot of work to be done to make that process a lot simpler and easier, but I think it will happen. And I think that the open standards part is absolutely fundamental.

Jules Urbach:

So I know, Patrick, of course, we're on the 3D formats working group working in glTF. I've been part of that, I think since 2016 or maybe earlier, and I love glTF because it is literally the JPEG of 3D. And of course, the web was... images on the web were JPEGs, it became [inaudible 00:17:53] later, but that's still largely the case. So when you're talking about something that's replicating the success or the value proposition of the open web, you need open standards for, at the very least, 3D models. And there's a number of different ways you can look at that, because even when you had and still have JPEGs, you still have Photoshop files, you have what goes into making that JPEG?

Jules Urbach:

And that's where I think the metaverse is a bit different than preceding platform ecosystems or even mediums. One of the things that I'm a big proponent of is I think every iteration of an asset or an item is, in some ways, almost equally valuable in the metaverse because you do want to have something that is almost like a JPEG to sort of view it or to have an iconic representation of an asset, but you'd also want to know essentially to the maximum level of its crisp reality, how that object for that asset was created, even how to render it, and that's where Render comes in.

Jules Urbach:

And for Render to work, even as a service, as a product for all of our customers, and we make plugins for 26 different 3D tools, I mean, even after effects is not... let's say only 3D tool has a plugin. We needed to come up with a way where we could take everything that was in the offering environment and save that into a format that we can then upload to the cloud, or now the decentralized blockchain, and not have after effects, and not have 3D engines or anything other than just the data, and then render it with just a standalone tool. And since then, we've also added other renders like Arnold, and Redshift, and even any Hydra Render Delegates, Unreal and Unity, all those things.

Jules Urbach:

So the concept of having a format for the absolute, the almost raw format or the archive format is really important. And we almost accidentally stumbled into such a format when we created ORBX, which was, I think back in like in 2012, or '13, or something. And it was really a virtual disc so that we could put anything in it, and everything that was in sort of the render state of any of these 3D applications. So every time a third party would add a plugin, if we did a Unity integration, which we did or Unreal plugin, you could take everything in Unreal, or Unity, or anything and it would save it as an Alembic file, or an FBX, or an OBJ. Those are the three mesh formats, EXRs, PNGs, and a bunch of XML with all the materials in a format that was suitable for Octane. But that itself was basically based on [inaudible 00:20:19] book on rendering, the laws of physics and light.

Jules Urbach:

So we had something that, for me, felt like, well, this is an asset that you could feed into the Star Trek holodeck for rendering. That was the goal, and Octane was, in some ways, the product to rendering that, but the metaverse can't be a product, or a company, or anything else, but because we'd already picked, as an export target for ORBX, open formats, and the wraparound it was pretty trivial, we started to get interest from groups like MPEG and JPEG back in the mid last decade. And I joined MPEG and JPEG just to see if we could take, ORBX and turn that into something that was an industry-wide spec.

Jules Urbach:

And I think it was so early when I was explaining, this is a scene graph of the metaverse, this is something where if you have a holographic display panel which partners like Light Field lab or building, this is what you would need because the Star Trek holodeck is actually like a goal that I want to see achieved and I want to have a format that can support it. And even for our company, I want data, and graphics and things to last decades. ORBX files created 10 years ago, still rendered just fine today, and now they can render the renders that didn't even exist when we launched Octane. So this is where ORBX came about as a spec, and it is a fairly heavy spec. I mean, it's meant to be lean because of heavy graphic software format, and it's evolved because it's essentially a scene graph and you can replace geometry with anything.

Jules Urbach:

We added glTF once glTF was created because it is an efficient way of storing models. We have multiple different image formats, but ultimately, they're transposable, and we've even done things where you can do take ORBX file and turn it into a glTF, turn it into a USD, or take glTF and USD files and put them inside of an ORBX, and they both work. That system is designed to be an umbrella so that we can catch every important piece of the authoring type [inaudible 00:22:03], at least, for rendering, and have that as a renderable format.

Jules Urbach:

And so ORBX, when we initially introduced some of these ideas to the glTF working group, I think the mandate then for glTF was, this needs to be really small and run-on mobile devices. And that is still correct, you want to have something that's really fast to transmit and load. And I was so impressed really with that mindset that I was like, "Well, we should adopt glTF inside of ORBX because it's so fast to load, it's so efficient, this is great. We want that as an option." And what's happening, I guess, now as glTF is sort of evolving is into things where people use it for 3D commerce, people are looking at it, frankly, a lot of 3D NFTs are glTF files that's dropped into a webpage.

Jules Urbach:

So I just tweeted about that this morning, what we want to do initially is just make it so that an ORBX file that's used to render, or create, or even export that glTF, and we can export and distill things down into glTF. Well, that has on chain, the full scene graph, and you could render the high-fidelity holographic version of it. And in some ways, from my perspective, that means that a glTF is that asset because when you look at how Render works, it's not storing in necessarily on an S3 bucket, all the textures, all the 3D models are [inaudible 00:23:15], ready to be looked up on IPFS or Arweave on the blockchain.

Jules Urbach:

We have an XML graph which we can pretty easily turn into JSON that just gives you instructions like an HML page of how this asset goes together. And what we do need is sort a link in the metaverse that can link to that. And my thinking is instead of having a text URL, you have a glTF file that has some sort of blockchain reference to the deeper asset or portal to it because we do need something like that, and that's kind of what we're experiment with. So it's kind of like by adding a bit of metadata into a glTF and dropping that in anything that can consume at glTF, any webpage, any market place, you have sort of the instructions like the Voyager Golden Record or the DNA for how this thing, what it means, what it's supposed to do.

Jules Urbach:

And the work that we've been doing with Render where we are rendering production renders on the blockchain, it's also... I mean, many NFTs are done on Render, and instead of giving people an image at the end of that, which is what a lot of NFTs are, the whole render job, the whole graph itself really is already an NFT in the sense that it's... The whole way that Render works is we have a proof of Render system if their render successfully completes, whether it's in real-time or whether it takes 10 hours, that's when the token exchange happen, that's where the compute resources used are paid out to the person that provided it and the artist, or creator, or consumer of the asset gets what they want.

Jules Urbach:

And that to me is also a fundamental part of the metaverse, because having something that proves what something is, and who created it, and how to recreate it, and how it can be remixed, and how those things could possibly interoperate is fundamental. And I think from a visual high quality rendering perspective, and possibly real-time, we're kind of there, but what's kind of missing is, well, what is the royalty stream for this thing, and what are the physics, and how does the interactivity work? And this is where things get much more complicated because I think that we're just starting to sort of see now inside of the USD spec, for example, Apple and Nvidia typically don't really agree on a lot of standards together, but in this case, they both settled on this rigid body physics schema for USD that's really good.

Jules Urbach:

We were thinking inside of ORBX, which is also now open source through CableLabs and ITMF plus add that schema, we support USD. So anything that USD has technically is inside of our larger framework, but this kind of standardization, even between things are inside of glTF, MaterialX, USD, for materials, for shaders, for physics, I mean, it's important because you don't want to have 20 different versions of it. And I think that that work is kind of happening. And I think that we're going to probably see some interesting convergence happening around MaterialX. I think the fact that that's now an academy software foundation project is really good.

Jules Urbach:

There's a MaterialX to ADL, to OSL and GOSL backend. So even if you're looking at one of those three as your shade or spec of choice, you have options there. When we added other renders to the render service other than Octane, we also gave something back. When we adopted Arnold standard surface, which the Autodesk team was trying to get out there as a spec, this frankly was better than the one we had in Octane. So that went into both ITMF and everything we're doing as well, and it's great. I mean, we are now much closer to having it so that if you pop up an Arnold or an OctaneRender, they will look the same with the same materials. And we're working with Maxon, doing the same with Redshift, and I think that's going to all sort of become a learning experience for us to really converge on, at least, for the highest possible quality, how this all works.

Jules Urbach:

For Unreal and Unity, I mean, Unity, we had this integration that was kind of bespoke for that back in 2017, and in Unreal, we launched that in 2019 where we can take an ORBX file and we can convert it to Unreal meshes, material shaders, HLSL, it's crazy, and back and forth. So we kind of are bringing all the benefits of all the open source pieces we've done, and we can transpose those into other formats. That transposability is fundamental to how a metaverse that needs to work. It shouldn't just be a format that is rigid, it should be the fact that that format is actually transposable into other formats that might have better utility for real-time, or for other purposes, or that hide or remove some data so that it can't be repurposed, but those are the kinds of things that start to make this very different.

Jules Urbach:

And even when you think about apps or games that are published with multiple backend, "Oh, this is for mobile. This is for web. This is for desktop. This is for console. This is for VR," what I think the metaverse truly needs is the meta version of the asset that basically allows a single sort of through line of virtual space, and gameplay, and whatever else you want to encompass to be rendered in these different modalities by creating it once and by having the system itself sort of help with these pieces.

Jules Urbach:

And ultimately, whether or not the world is going to be rendered in two shading or completely similar to the world we live in with photorealistic lighting and rendering, the idea of assets themselves kind of being certainly from the author and the creative perspective defined first, and then having those pieces added is a really another fundamental piece. And that's why OTOY as a company, for me, our role is for artists, not necessarily the tools for the software, the product, or the service, we are here to serve artists so that their creations basically continue, and exist, and interact in all the ways in metaverse will allow. And so, these are just conceptually the things that are in my head every day as I'm thinking about all these pieces coming together.

Patrick Cozzi:

So Jules, I love the philosophy, I love the vision, I love your passion for this. I mean, do you have a call to action, especially around the open standards? What should we, as a community... what's the next step? What's the next experimentation? What should we be doing?

Jules Urbach:

I think that it would be amazing if you had multiple... I mean, listen, as a company, obviously, I make a rendered product that I sell, so having other renders on the service that we also created it, surprises a lot of people. I think it would be amazing if you could, for example, get some sort of convergence between glTF and USD, we're doing that in ITMF. ITMF exists, ORBX exists to really smooth out those differences. But I do think that, if you could have protocols that could allow whether something's in USD or glTF format, or any other format, that kind of exist in a scene graph that is so obvious in how it's represented that no one argues with it. I mean, that's kind of where an HTML page kind of exists. You can go to a webpage and at the very least, you'll have a body, and a header, and a favicon and things like that. I mean, some sort of high-level framework for any of 3D spec or future specs or future iterations is important and that's philosophically where the ITMF... Not the full package part, but just the graph of it.

Jules Urbach:

The equivalent of HTML is something that should just be a JSON graph table, that's fundamental. And then you want to have, at the very least, the blockchains themselves no matter which chain you're using, no matter what marketplace it's on, you have some way of tagging the commerce aspect because obviously making money and selling things and NFT is already a really important piece of that. And I think those are some of the fundamental pieces that you attach that to really healthy standards that are already strong, like USD and glTF and anything we can add. I don't care if ITMF ORBX ever gets adopted, I just want those features to be available broadly in whatever the metaverse becomes.

Jules Urbach:

So in some sense, as long as the framework that everyone's sort of going after is kind of defined in well, whatever we have in a webpage, whether it's a JPEG, a PNG, WebP, or some other thing, the image element is defined, the [inaudible 00:31:02] is defined, and there's maybe some system where Rust code on the GPU can actually be its own renderer or formats themselves can be self-describing or procedural through a system that is the equivalent of WebAssembly or JavaScript. Potentially, that, to me, feels like that framework should be a priority and the entire industry needs to agree that that's an important piece.

Jules Urbach:

My concern is that if you have the mindset of like, "Well, if it's not on my device or it's not with my sign-in, something fundamentally can't be taken out of my bubble from a... I created in this bubble, I'm taking it out of it." That doesn't feel right to me. You need to have the equivalent of identity interoperability, asset interoperability, not just at a technology or data level, but at a... Like, "This can leave, I can take this elsewhere."

Jules Urbach:

Some of this has actually kind of happened with Disney doing the movies [inaudible 00:31:54] thing, you buy something on iTunes, or Google Play, you kind of own the movie, you can watch it on the other one, not perfect, but that should be a fundamental thing of ownership and art and things that are happening in one section of the metaverse, no matter what it is, the actual asset or idea of itself is transposable.

Jules Urbach:

And that's where we've seen all these mistakes happen, where, if your identity is locked in to one social network, or your data is for that matter or how that data is used, it's not great. So I do feel like that's where having things clearly defined early on and ensuring that nobody's trying to put those in place and lock the rest of the industry out for doing something about it, I think is pretty important. So, yeah, I mean, that's how I see some of the important red flags of early metaverse adoption, needing to be really well thought out, really well thought out.

Marc Petit:

Can you highlight for us what you did with ITMF? You made some interesting inference about being a super format and scene representation, I think would be interesting to dig a little bit in there.

Jules Urbach:

Yeah, so ITMF literally came out of this Darwinian evolution of having to support every 3D tool because it wasn't just us making integrations [inaudible 00:33:07], It was every third party that came to us, says, "Hey, we'd like to license and create a plugin for Autodesk Inventor and SketchUp. I mean, that's how we got to practically 30 tools and this has been the case forever. I mean, OTOY has always been a cloud-centric company. Cloud has always been an important vision. I had the idea for distributed render for a while, and I knew for a fact that if I'm rendering on a Linux machine and C40, for example, didn't have C40 for Linux, so I needed a way to take out everything from the 3D tool and export it. And I figured I could have done this in a proprietary format, or I forward as it turned out, the [inaudible 00:33:41] was just useful enough at the time, to be a pretty good target for animation. I mean, now you lose things with it, you don't have scalable deformations, USD adds that.

Jules Urbach:

So inside of ITMF, ITMF is the open-source version of ORBX, like ORBX, OTOY created it, it's there, it's very much like what Apple created, QuickTime, the MOV format and then gave that to MPEG and MP4 and MOV are practically the same, but MP4 is the MPEG industry adopted version of that. And ITMF is equivalent relative to ORBX. I mean, right now they're pretty much identical. We're helping to keep the ITMF going. We've got other partners in IDEA and IDEA's very focused on this sort of master level like we're going for the holodeck, that's what the mandate for that working group was. But we see the fact that glTF and USD are both sub-assets in this thing, is important. And also, right now within the glTF working group, we're proposing, "Let's just take the graph part of ITMF, and at least see what parts of that can make sense inside of a glTF file or even in USD."

Jules Urbach:

There's nothing wrong with these things going into other formats, and similarly, in some ways, because of that, things that are added in USD, like the physics system, and they have an audio thing in USD, these are elements that were sort of like, "Well, this goes into ITMF and this could also then go back into a glTF." So ITMF really is something that... I don't know of any other format that's out there that allows you to take your art, which is what I cared about when the [inaudible 00:35:04] was created and render it without the DCC, with everything there. There's no plug-ins, you don't need to worry about what the DCC was, what the plug-ins are. And the last step was, we need to get rid of our own renderer and make it so that you can load in any renderer and render an ITMF file and we now have at least three. We've got Mars, Venus, and Earth, in the sense of Redshift, Arnold, and Octane are all kind of able to transpose these for adding cycles, first open-source, full renderer, and this all operating also with any Hydra Render Delegates.

Jules Urbach:

So we did this crazy thing where we can turn ORBX or an ITMF into a USD scene delegate, Hydra Render Delegate, and then any render delegate can connect to that and it's really interesting. So by doing so, we've sort of blown open all these weird doors, and we even have a streaming spec where you could take one endpoint that's generating an ITMF or stream and send it to another and mix those together. So we could cross the stream, do we have Unreal and Unity both running, and then both mixing those things together in either thing. And that is a bit of what Omniverse is doing as well with the connector system. And I love the Omniverse team, and I know Richard Kerris really well. And obviously, USD is a big part of that pipeline. And so, we're building connectors to Omniverse. But if you think about the larger metaverse, I mean, you need something that runs potentially not on a Nvidia hardware.

Jules Urbach:

The thing is that the three renders... I guess the two renderers that are at Omniverse are render delegates but they don't run in anything but Nvidia cards, I think, and also there is no real Mac client for omniverse. And more importantly, I'm thinking that's not the real deal-breaker, we just need something larger that kind of smooths at those edges. So if you wanted to stick to that system, it still works in the larger metaverse, the data can go out of it, into it, it's not locked into omniverse or ITMF, whatever. ITMF is ultimately, I just gave that to CableLabs and there's no royalty, there's no nothing, all the FP is tier one. I couldn't do that in MPEG, by the way, that's one of the reasons why I stopped focusing on doing this through MPEG. And CableLabs which had basically been a big part of the MPEG working group, helped to start IDEA with people that were like-minded. And all of that to me is just knowledge to be donated to anyone else interested in Open Metaverse.

Jules Urbach:

So you can take pieces or good ideas in ITMF, and you can put them in other things, but ultimately, as I'm part of other consortiums and working groups like glTF, I'm thinking, a JSON profile of ITMF inside of glTF could be super useful. And that's exactly the kind of thinking that I think will lead to an Open Metaverse spec. But maybe ITMF's the one, it's simple enough, it's already mostly an umbrella format in JSON or XML, wrapping around other things, and it's always changing with other parts of the industry. If you think about how OSL, USD, EXR, glTF, in themselves evolve, they are evolving the ITMF format in and of themselves. So it's a very intriguing option and I think because of the fact that we have this amazing opportunity at the ground floor, with the Solana team, they're recent blockchain, they're basically the number five blockchain by market cap. And they are on Ethereum and NFTs are mostly on Ethereum, but Solana is much faster transaction volumes. There's a bridge that goes back to Ethereum, so you don't lose any continuity there.

Jules Urbach:

And as I'm thinking of storing things on-chain for real-time, scene graph changes, I need something faster. And we looked at a brand, we looked at everything else, the Solana framework seems really intriguing. You can use credit card payments on Solana, which is really important for crypto users that don't know crypto that well yet and they're also starting a whole NFT spec that is just very open. So they're like, "What do you guys want? And what can we do to help?" And I'm thinking, "Well, let's get glTF with the ITMF extras in there because they already have on Metaplex, which is their baseline equivalent of the ERC-721 NFT, the ability to drop into glTF file, I just want to extend that to streamable portals and full scene graphs. And I think that if we can prove that on Solana and we can prove it with the glTF and the ITMF pieces in there and add all those elements in the next few months, well, then we've got a proof of concept where other chains and other... I mean, we can at least start figuring out what went wrong or right with that, and then use that as a pretty good starting point for a larger, real industry standard. And I love that. I mean, that's why this year is so exciting, super exciting.

Marc Petit:

Yeah, and now, look, I heard you talk about Solana and starting to mix real-time and chain in the same sentences and there was kind of starting to raise my [crosstalk 00:39:31] interest.

Jules Urbach:

[crosstalk 00:39:32]

Marc Petit:

Yeah. Explain to us a little bit more because, okay, Render Network has a way to delegate offline rendering, but you starting to use sentence like, "You render on the chain or the data is on the Render Network." And so, you seem to imply render on demand even though the way to real-time rendering and interactive content on the Render Network, so is your implementation on the networks going all the way there?

Jules Urbach:

Yes. Here's the interesting thing is if you look at the two primary real-time engines are Unity and Unreal, which we've had a strong and long and deep relationship with both of them, and with Epic as well and we've done so many crazy things. We've been doing streaming and cloud gaming since before Gaikai and OnLive. We were doing this in 2006 and seven, eight. So for me, streaming from the cloud is, yeah... I mean, it's like we just switched from cloud gaming to doing... Autodesk is our largest investor, earliest investor and they invested in us because OnLive, I guess, was at some point, doing their app streaming and then we just did their app streaming and it worked great. That's how we got to know Mozilla and Amazon and we helped launch the G2 instance in 2013 with all of them.

Jules Urbach:

So for me, streaming from the cloud is just a GPU cost. Basically, if you're building up a GPU on Amazon to stream anything, you're still paying for those GPU minutes. If it's an offline render, you're paying for those GPU minutes in [inaudible 00:41:05], because it's a high latency job. But imagine that you could do the job in real-time at 60 frames a second with just more GPUs like Octane scales, we gave 100 or 1000, we showed this in 2013. Jensen brought me on stage and we 100 GPUs and it practically was real-time. Now, it only takes five GPUs, because this is 2013. But the idea of render and the OctaneBench score, or the compute cost of a render is something that you could just add more power to get it to be in real-time or simplify certain steps in the rendering process but you're still paying for GPU cost somewhere. And that doesn't go away if you're doing anything other than local rendering.

Jules Urbach:

Now maybe one GPU locally is all you might need to render everything, but what if the asset itself is 100 gigabytes and then you have to deal with a download cost. And what if the 100 gigabytes need to be touched by 1000 other people, maybe it needs to be stored somewhere else. So the way that I see it is we're doing this in the right order. I wanted to get offline rendering to work well, that's never going away because as much as you think real-time rendering has solved the problem with 4K or 8K, well, nowhere come holographic displays. That's, basically, 10 gigapixels per square meter, and you need 16 GPUs per foot, to just power the panel to display it. 16 is 6,000, by the way, so it's insane the power consumption. So we're not done with offline at all, that's pre-computed rendering for those types of devices, for high-resolution displays are always a thing, there's always a higher quality, maybe one day real-time, but there's a spectrum.

Jules Urbach:

And so, as we look at how NFTs are evolving, there are people that have minted and NFTs, because, on Render, they can render 5,000 versions of the same image with slightly different materials and some beautiful output and render does that really easily. The way towards real-time though is, essentially, getting back to job instantly is real time.

Jules Urbach:

So in other words, if you have your scene, what we do is we atomize the whole thing, it's not an ORBX when it's on the Render Network, it's atomized into SHA-256 Hash for every single file. We don't know the file name anymore, we just know if it's uniquely been uploaded and that's exactly how distributed file storage on the blockchain works. It also maps back to S3. So we just have the literal DB, the information theory hash of the model or texture, and then we have a graph, including all the delta, all the changes, as you recompose it and rebuild it. So if Beeple wants to rebuild his NFTs that he sold at [inaudible 00:43:22] resolution, we can do that on Render, we've got all these things on there, and now we have the ability, I mean with Epic 4.27 has the ability to take the entire Unreal Engine and put that into a module. It's open source. We license to Cinema 4D renders open.

Jules Urbach:

So we don't need to actually even export to ORBX, we can put the source files there, and those become one option. And ultimately, what we want to be able to do is give you the option of saying, I want an offline renderer. And some of those things have to happen because it's not a real time thing at all. It's just like this needs to go on a holographic display, or this needs to be a piece of art that is fixed, and that's it, or I want the same thing in real-time and I'm willing to lose the quality because we have Brigade and we even have... We can transpose ITMF into Unreal, or Unity, or any other hyper augmented reality and give you back a live stream. And in order to pay for them on the Render Network, we will pay a premium if you just keep your GPU running, just ready for a live stream in any moment. Because a lot of how render works is that, if you're using your GPU, it's not available on render. In order to be ready for live streaming, we need to have that availability.

Jules Urbach:

But we have such a diffused number of nodes all over the world that even latency becomes much simpler than traditional cloud gaming or traditional data centers because the first week of render, we have 1200 different nodes all over the map with an average of 1080 PIs each, which gives massive amounts of power, but also massive amounts of distribution. So we probably can get to endpoints or the edge pretty easily with streaming on render, and we're just going to make it so that instead of just uploading art, you can now upload, at least from Unreal and Unity and maybe what we're doing within glTF or USD Physics, just something that is viewable and it's not meant to be any specific thing. I mean, it's meant to be this large larger framework for just giving you the ability to do realtime computing and offline computing with the same service. And you're still paying for compute cycles either way, probably, in some ways, even ray facing cycles, as real time evolves towards real time ray tracing.

Patrick Cozzi:

So Jules, what about network bandwidth for real time, say for mobile, is 5G enough or how far does network bandwidth need to go?

Jules Urbach:

It's fine right now, frankly, for anything that isn't AR or let's call it XR. And even there, it's interesting because you have a Nvidia's entire CloudXR stack, which is a method that I think works pretty well. If you're talking about AR and VR in particular, you already have positional time warping that was largely pushed forward by the Oculus team, and [inaudible 00:45:46], and others, where it takes the depth buffer and it reprojects it so you have fluid motion even if the render, locally, is too slow.

Jules Urbach:

So in some sense, latency, if you have positional time warping, is really well handled, state information for how that world's rendered is no different than if you're playing on a server where the rendering is local but the world's information is coming in from the cloud. So I think that 5G is fine, I think it's fine for everything up into super high 4K or 8K display that might be in each eye. But I think that in some ways, AR and VR rendering, because you're obviously going to be streaming the [inaudible 00:46:18] and all those other things, it could work. The tricky part is when you have mixed reality and you have the world locally being mixed with something that's a cloud stream. And we've been playing with that too. It's not impossible, but you're just basically doing a bidirectional scan or stream of the world.

Jules Urbach:

And I think that we're at a point now where even Octane runs just fine in on the iPhone 11, it's not super fast, but it is faster than the GPUs that were on Amazon in the G2 instances in 2013, which is crazy. That was the initial cloud gaming. The great K520 are now slower than an iPhone 11 and Octane runs on both. So we're at a point where, yes, we can render things locally and beautifully. But I do think that having the ability to sort of scan the environment, put that into the device, and also send that to the cloud, and coordinate those things, I think that can happen over 5G.

Jules Urbach:

I mean, when you look at what 6G's trying to do, it's like that's where you're just getting into the ability to have shared GPU memory or something equivalent to that over everything. Now, there is a case for that with holographic display panels, and things that are going into the gigapixel resolution, which will be video walls of the future, possibly every a 100 inch TV in a home in the 2030s will have that resolution of 4,000 DPI and all this stuff. And that's what CableLabs was looking for ITMF to help provide a standard so they could justify a 100 gig of the fiber to the home, and potentially, 6G would have the same requirements. But for, I think, the glasses and devices we have today, 5G is pretty good, it's really good. If you have the coverage, obviously, that's 5G coverage, not so great.

Marc Petit:

So I wanted to go check on something else. I mean, on top of everything you described about a [inaudible 00:47:54] rendering, OTOY is involved in light field rendering and pioneering of that space. Can you speak to that, and when can we... You kind of touched [inaudible 00:48:06] on the amount of compute that's necessary to make light field rendering a reality, how do we think about light field rendering?

Jules Urbach:

So I think of it very much... and I think a lot of this is defined really by the end point. So when, when Oculus was a thing, and the iPhone became a thing, everything changed around that, in the sense of you started to have the idea of a VR platform or a mobile platform. So for me, light fields were... it's like there's so many different ways you can look at it. I look at it very much as like, well, I want the Star Trek holodeck, that is the ultimate endpoint for mixed reality or experience.

Jules Urbach:

I don't like wearing glasses. I mean, I don't think most people like wearing even simple polarized glasses because otherwise 3D TV would be a thing. So you know, for a fact that nobody wants to wear anything if they could avoid it. And just thinking through that to the end point, the holodeck, those that aren't Star Trek fans or know about it, it's a room on the Starship Enterprise that you go into and it creates any environment for you. It's a small room, it's not an infinitely large, but it feels like you're there and you can interact with it, and it even has this kind of treadmill effect where you keep walking, and you think you're always in this infinitely large simulated reality.

Jules Urbach:

Those devices, it's funny with Star Trek or some of these things that are hundreds of years out happened in 10 years, Picard had this thing, this iPad, that was so advanced in the 90s and then 2011, Steve Jobs roll out the iPad. We're at the point, in some ways, with holographic display panels. Those, to me, are the... every time that I would say about light fields the last 20 years I was thinking about, at some point, the resolution will get so good that you'll be able to replicate a hologram. And a hologram digital or whether it's on film, is still basically creating... essentially for every tiny point it's creating a light field that you see and that feels like you're looking at something real.

Jules Urbach:

When you look out of a window, there's a barrier between you and the rest of the world but the reason why you experience reality on the other side of the window is there's a light field propagating through the glass. And if that glass was an emissive display, and you could replace a window with a light field panel, you might not care what view you have in your apartment. You might actually be able to live on Mars or on the moon or just experience metaverse like you would experience reality, at least visually.

Jules Urbach:

And I was always thinking like, "This is going to happen in 2030, I hope I'm live when these panels happen." And as it turned out, a team that worked initially at Lytro and for RealD. My friend and genius, Jon Karafin and his team, they had a startup called Light Field Lab. I immediately was like, "I want to be an advisor. I want to help you guys get going." They've got tons of funding, Samsung, I think, and others invested about 28 million in them. And they just showed their first light field panel. And our partnership with them is all about taking renders and rendering to those panels, that is our grand truth for knowing whether or not light field rendering that we've been working on forever even has any meaning.

Jules Urbach:

And so we're starting of course with offline stuff, but the idea is you can also play blast light field animations, or content, or even do the equivalent of QuickTime object movies, where you can move the Starship Enterprise [inaudible 00:51:08], which is another test case for us. So the idea of these holographic display panels are very much why, for me, light fields, whether you're talking about it as a format or as a concept, is inescapable. And I would say that that's not just going to be for large wall size display panels, eventually, if you have any screens, they will be light field screens. There's no doubt that that's going to become another obvious thing because, why not?

Jules Urbach:

I think at CS, you just had, I forget it was Acer, one of the laptop companies that basically showed something with [inaudible 00:51:36] where you had a traditional lenticulus thing, but it has eye tracking. So you see something that looks holographic, that's still really compelling, but imagine that on your desk and everything else, and I feel like that is going to drive so many of the experiences we have in the metaverse. The glasses may be an alternative to that or parallel track, but I do think that that's one of the reasons why light fields are inescapable, and from a, I want them something to look real, and feel real, and be real, I mean, that's kind of like the defraction limit of light, where that resolution starts to matter. That's where having something that is renderable at that fidelity becomes important.

Jules Urbach:

And of course, then it's like, "Well, [inaudible 00:52:12] we make this interactive, can you add touch? Can you make it so it's physics," all that stuff. But those are layers that we can get to, certainly, once all the other elements are starting to coalesce around the graphic stack itself, but they are working on Ultrahaptics for touch, for example, and the first display panels we're going into production this year. So Beeple wanted one for his physical NFT to display his renders on. And I just think this may have went to the museums like the [inaudible 00:52:35] eventually, and then theme parks, and eventually a 100 inch TVs, and windows, and maybe entire buildings will be covered in holographic panels and cities will look totally different because you can make a building invisible or you can make it look like anything. Super interesting stuff.

Jules Urbach:

And that's where I've always been intrigued by that, the holographic display has been, since I was eight years old, something I've fundamentally wanted to see happen in my lifetime, and here we are. So it's a big part of why the light field content pipeline has always been super important.

Marc Petit:

10 years out.

Jules Urbach:

Oh, I think you're going to be able to pay for it this year, if you really want one of these things in your theme park, for your concert. Those 20 inch panels, the one that they showed a few months back, publicly, [inaudible 00:53:22] they invited the press... It's like the Samsung Video Wall, you just buy another 20 inch or 20, and you can make it as big as you want. The larger the holographic panel, the further in and out holographic objects can be pushed and you can also then turn it into a wall or ceiling or floor, and you can build a holodeck. It's like tens of millions of dollars to do that now. And I kind of feel like you probably want to wait at generation or two, when they get to scale, it becomes way cheaper.

Jules Urbach:

But it's going to be like OLED, it's not going to be anything fancy, that's when the scale becomes universal. So this is the decade where I think you'll start to see that at the high end, and then in 2030s, you'll see that in homes and in devices everywhere. And then the capturing side of it and the content side becomes interesting too, because even if you're thinking about this as a 100 inch holographic TV, is that going to be left and right eye depth maps? No. Real-time rendering? Who knows? Maybe the way that I see that working is we have this interesting metaverse-ready project. I like picking something that is creatively an art project for us at OTOY to do, because it helps us dog food our own ideas. I created a company as an artist I created to try to get... because I wanted to make things. I'm a gamer, and I'm an artist, and I love that.

Jules Urbach:

So my best friend, Zach, created Star Trek, is an investor in OTOY, one of the projects that we've been doing for years now has been to basically archive all of Gene Roddenberry work, and that include these blueprints of the Enterprise and Roddenberry, my best friend. And as we were getting into the last few years of this project, he's like, let's consider building the Enterprise [inaudible 00:54:57]. They almost did that in Las Vegas in the '90s, built a life-size Enterprise you could visit his hotel.

Jules Urbach:

And we've done that, we've showed that. It actually was featured in the Apple event. They were showing what you did with 64 gig GPU and this Enterprise model that's been put together by the people that worked on the show with details, and interiors, and everything working, that's a metaverse asset, and we're at the point now where we have all the different versions of the ship. We have every single episode, where is the ship? Where is the scene? Where is the camera? We can practically render the episodes to the end, we're now getting to the point where people like Spark, and Kirk, and stuff are renderable. And I'm thinking, if we could basically figure it out between AI capture and artistry, how to tell any episode of TV or any event ever, and we'll start with just the sort of meta history of the Starship Enterprise, then we've got a pretty good sense of how content and certainly, experiences that are telling a story might work at a meta level, and I love that idea.

Jules Urbach:

So just getting the Starship Enterprise right to make Gene Roddenberry proud, in the metaverse, if we can get that one thing, that one asset right, and figure out all the pieces that connect to it, then I think there's as much more tractable proposition than building the open metaverse or looking at the entire thing, this is something that we're getting so close to holographically rendering beautiful Star Trek scenes on a holographic display panel that I'm like, holy moly. And I think AI is helping us actually get... be able to track what actors had done on older date shots, and then turn that into 3D and render it again. And it's wild, that kind of stuff, it's amazing. It's amazing.

Marc Petit:

Fantastic, fascinating. Jules, thank you so much, Patrick. I think we're going to go to two last question with usually how we close those shows. First. I know we've talked a lot of things, but any other topic that you had expect to cover and we did not cover today.

Patrick Cozzi:

I think we covered a lot.

Marc Petit:

Yeah. Usually, I tell people, when you listen to a podcast, you can listen that 1.25 speed so that it goes faster, I don't recommend to listen to you at 1.25 speed. It was so much to unpack and to... thank you so much. Another thing, you mentioned some names, but any organizational or person that you want to give a shout out to that can play or has played an important role in thinking about the Open Metaverse?

Jules Urbach:

Ari Emanuel is my brother from another mother. He's the CEO of Endeavor. If you ever watch the show Entourage the Ari Gold character is based on him. He helped me build this company. And he's been absolutely an amazing collaborator. I mean, I wouldn't be here if it weren't for him. Yeah, he is my brother from another mother. And many others, I mean, I have an incredible team at OTOY.

Marc Petit:

You have incredible advisors too, congratulations on your ability too-

Jules Urbach:

And incredible advisors too, that is correct. I'm surrounded by just incredible people. And yeah, and it's a team effort.

Marc Petit:

Wonderful. We're super lucky to have you today. So Jules, thank you for your generosity.

Patrick Cozzi:

Yeah. Thank you so much, Jules.

Jules Urbach:

Thank you, Patrick. Thank you, Marc. It was such a pleasure.

Marc Petit:

I have homework to do with ITMF and glTF because I took notes on that and I want to do some digging there. It was very intriguing and we're going to look into this. So Jules, thank you so much. Thank you to everybody who's listening to the podcast. We're lucky the metaverse, it's a trendy topic so we have a lot of interest, we got some accolade and recognition, so thank you everybody. Please review, let us know who you want to see on the show. Let us know how we can make it better. Bye everybody.