The Evolution of Computer Graphics

Announcer:

Today on Building The Open Metaverse.

Eric Haines:

So I think that changing things in the virtual world thing is significant. It's nice to have the, "Well, we went to the cool music event or whatever, or the party and we met up." And that's fine to have a social space and so on. But I think that virtual thing where you're actually building something and that it has some kind of permanence in a sense, there's really something strong about that, there's something really compelling about that.

Announcer:

Welcome to Building The Open Metaverse, where technology experts discuss how the community is building the open metaverse together. Hosted by Patrick Cozzi from Cesium and Marc Petit from Epic Games.

Patrick Cozzi:

Hello everyone and welcome to our show, Building The Open Metaverse, the podcast where technologists share their insights on how their community is building the metaverse together. I'm Patrick Cozzi from Cesium. Our co-host is Mark Petit from Epic Games. He is out this week, but he's with us in spirit. Today we have a very special guest that's going to be a real treat. With us today is Eric Haines, distinguished engineer at NVIDIA. Eric, how's it going?

Eric Haines:

Great.

Patrick Cozzi:

So Eric, you've done many things in the computer graphics community over your career thus far. A few that I really appreciate, the four additions of the real-time rendering book as co-author. I think I learned as much from your books than I did in grad school. And I learned a lot in grad school about computer graphics. You've done so much else to support the community, your Udacity course for interactive 3D graphics- I think it reached so many folks, Ray Tracing News. I remember reading many of those articles and implementing a few of them as well. And more recently, Ray Tracing Gems, everything you've done for the I3D conference. And then thinking of the metaverse with Minecraft, what you did with Mineways for 3D export to do 3D printing or to publish out to Sketchfab. And then your work now at NVIDIA with Omniverse. I could go on and on, but we'd love to hear your journey to the metaverse from your perspective.

Eric Haines:

Yeah. It's funny because in preparation for this podcast, I had been sort of thinking about what my history as it were. And yeah, so I started out in high school in 1974 or something. There was a PDP 11 computer with all of 4K. It had little blinking lights on the front. You could toggle in your instructions that way. But there was... oh, okay, now bad things are happening on my screen. Discord. I forgot to turn off Discord. But anyway, yeah. So right. Had blinking lights, you had all 4K memory. You would load up a compiler... or an interpreter for BASIC or whatever. And that'd be three of your 4K. You had a little teletype for your output. I mean, it was nothing.

Eric Haines:

But nonetheless you could do kind of cool stuff. You could play games. If you sent the right kinds of loops to your computer, you could make noises coming out of... your transistor radio could pick it up and just crazy things like that. Then I got to use the computer, the school computer, the data processing one- than I got to use a Fortran and I had punch cards and all that kind of junk. Right? So this is basically old paper tape punch cards kind of day. But what was interesting to me was that back then even then I sort of had this urge to share, which was just I had... So I translated a bunch of these BASIC games written in the language BASIC to Fortran. And so I wrote a letter to the People's Computer Company Newspaper, which was just an enthusiast kind of newspaper saying, "Hey, I've done this translation and here's my address. If you want them, I'll send you some listings."

Eric Haines:

And I didn't think I'd get any responses, but I got about 50 responses of people going, "Hey, send me listings." And so I still remember printing out a whole bunch of these and sitting on the floor with a friend and cutting them out and putting them in envelopes and sending them off as postage and all this. And it was just such a... You think about that now and it's like, "Wow, that's so crazy." We used mail. We had to go to the library if you wanted to look up a paper. There's just all this stuff that... I remember the internet year I remember, oh, you there's sort of you could put things on Usenet and so on and in the '80s, and that was cool.

Eric Haines:

And then seeing, oh, my gosh, the web, the sort of a web browser kind of thing in the '90s. That was just another miracle. I mean, that's just been great sort of about things in general. It's obviously information sharing's just been getting easier and easier, and that's all been fine. But also what's great is just to see how graphics has kind of evolved. I mean, really graphics I think, well in college I sort of started with punched cards, and by the end of college at the graphics lab at the university I was at Rensselaer Polytechnic, they had this networked system of sort of vector displays which were interactive speed vector displays.

Eric Haines:

And of course I didn't use them for any useful work and nor did my friends. What we did was one guy had invented this game called Tank War. And of course we played the living daylights out of that, and utterly addictive and you're like, "Oh, my gosh this is a glimpse of the future that this interactive thing in 1980 - Wow, if we can make this happen, this is just incredible."

Eric Haines:

Anyway, so yeah, so after I graduated in '80 I worked for a satellite launch company actually for a few years, RCA Astro Electronics in Princeton, New Jersey. And that was fun. But due to my sort of undergrad years, I had met this guy named Al Barr who's at Caltech now. And he is one of these graphics pioneer types. And I just remember visiting him in Harvard Square in 1982, sitting in his unfurnished apartment, basically sitting on the floor, looking at this thing called SIGGRAPH and looking at the back cover of that and going, "Wow, what the heck? All these cool things are going on. I got to get into that." So he told me the universities to apply to, and I got into Cornell and I went to Cornell and did that for a few years.

Eric Haines:

And that's where I learned about ray tracing and did some ray tracing research there. I was there at the time... I can't say I had any part of it, but I was there at the time the Cornell box, which is this famous computer graphics sort of cool little test scene appeared that... It's funny. Don Greenberg, the guy ran the lab, told us, well told our group basically, "Go make a box and paint a side red and paint the other side blue." And they're like, "What?" And he came back a week later he's like, "Well, where's my box?" And they're like, "You mean you wanted to make this real box?" So it was just that kind of... But I love that idea of, okay, well, can we make something real and can we actually really simulate it in the virtual world.

Eric Haines:

And I'm still doing that today. I mean, I've been sort of going back and forth with people about, well, can we just get a cheap light meter and a light bulb and just calculate exactly what we should be getting and how is our physically based rendering actually comparing to that? Very simple experiments, probably done all the time in the lab, but now for 50 bucks on Amazon you can buy all the equipment you need, which two decades ago it might be $100,000 dollars to get that equipment. And that's sort of another recurring theme.

Eric Haines:

So anyway, after Cornell I went into... My advisor basically became my president. We had a little startup called 3D/EYE, and there I got involved in just ray tracing for Hewlett-Packard. They were coming out with stuff for mechanical parts and so on, and they wanted to make cool pictures. And so we did ray tracing, radiosity, and stuff for them. And at the same time, I also got involved with Andrew Glassner, invited me to work on a ray tracing course for SIGGRAPH. And that evolved into the first book about ray tracing, Introduction to Ray Tracing, which you can now download for free I'm happy to say. He was able to sort of free it from the publisher and now it's free to download.

Eric Haines:

And it's still valid. Math doesn't rot. That's the fun thing about textbooks and so on is that that math it's not like something broke and we've all decided two plus two equals five. Those are all still fine equations. But anyway, so for me one of the amazing experiences was that... So in 1987 I was working on a bunch of little databases, little procedural kinds of things like, “here let's just have a bunch of spheres that are kind of in a sort of fractally pattern.” And I call that one “sphere flake.” That had kind of a cool name. But just a bunch of these sort of test pattern kind of things because I wanted to test ray tracers. We were trying to sort of compare and go, well, what's a good algorithm, what's the fastest way to do this, that, or the other?

Eric Haines:

And so I posted that code to what was before web browsers and so on, this thing called Usenet. I just posted the code about three weeks before SIGGRAPH and I thought nothing more of it. I'm like, "Yeah, maybe someone will use it." And then I went to SIGGRAPH and on the show floor AT&T Pixel Machines had implemented, had actually taken my code and made it happen. And it was a revelation. They had this $150,000 machine that basically could ray trace this thing, this sphere flake image, which was just 8,000 spheres, reflective spheres with three lights, nothing too complicated by today's standards. But back then it took me two hours and a good expensive workstation to render that image. Now they're doing it in 30 seconds.

Eric Haines:

And it was also the first interactive ray tracer that you move a mouse around, move this little shiny sphere on a plane and it would reflect at then three frames per second or something. Anyway, so that to me was just okay, with enough hardware you can make this thing go and it's just a matter of time. And so now we're kind of here. I mean really in the past five years NVIDIA has introduced ray tracing hardware in that part of their GPU, just an extra thing on the GPU. AMD is following suit, Intel, Sony. A lot of vendors are basically going, "Yeah, how can we do this? This is cool. What's it all about?"

Patrick Cozzi:

Absolutely. Look, so much exciting stuff that you're sharing. And I love that you had sharing in your DNA from the beginning of your time in graphics and computing. And I think it's people like you that have made the whole community very collaborative. I love all the tie-ins with SIGGRAPH. I think you know this podcast actually started from the Building the Open Metaverse BOF at SIGGRAPH last year. And we had so much to talk about, we started this podcast. And the story of 1982, looking at the back of that SIGGRAPH journal I think is really cool. So Eric, what does the metaverse mean to you?

Eric Haines:

Yeah. Right. I mean, that's one of those things, right? Where it's, well, what does it mean? And what it means to me is, well, it can mean a lot of things really. One can just simply be NVIDIA, their interest in the metaverse is things like digital twins where you say, okay, I'm going to make a warehouse and I'll be able to simulate that warehouse as fully as possible, and I can train robots on it or whatever. And I can basically simulate things so that I'll know if it's dangerous to workers or anything like that. And certainly I've left out bits of my history, but for 20 years I worked at Autodesk, and one of their ideas was with Revit is just having this sort of living version of the building.

Eric Haines:

It's like, well, you built the building and architects plans are not necessarily how the building actually gets built in the field. And so you sort of have as-built but you also want to try to, if you can, record that data. You can sort of say, "Oh, well we did this slightly different thing here for whatever reason." Okay. Well, now you know that's part of the living building plan and you can actually then use that living building to do other things like maintenance or just paying attention to a lot of different... or just knowing what's behind a wall. I mean, a lot of times you'll just have a plan of something from who knows when.

Eric Haines:

I live in a hundred and twenty, thirty year old house. I don't know, what's behind most of the walls, and it's usually a revelation. You take off some shingles and go, "Oh, dear. That's going to cost a lot of money." So having that kind of living, building thing and knowing, well, when was the installation put in, where were things done. So that kind of thing where sort of the virtual and the real overlap, I think, is a really interesting space.

Eric Haines:

There's also sort of the metaverse of just games of... We've certainly seen Fortnite or World of Warcraft MMOs kind of thing. But yeah, I mean, you mentioned Minecraft and that one to me really addictive because it was permanent in the sense that you would do something in that world and that changed the world. It wasn't like world of Warcraft. Well, World of Warcraft, "Oh, I just killed the cool monster. Oh, don't worry. In 15 minutes, he'll be back." And you've basically made no physical changes or virtual changes in that world. Right?

Eric Haines:

So I think that changing things in the virtual world thing is significant. It's nice to have the, "Well, we went to the cool music event or whatever, or the party, and we met up." And that's fine to have these social spaces and so on, but I think that virtual thing where you're actually building something, that it has some kind of permanence in a sense. There's really something strong about that. There's something really compelling about that. I admitted I'm doing it tonight. And I'm meeting up with my two sons and we're going to Valheim. And it's fun. You're sort of building things and you're exploring and you're just doing it together. And that to me, that's a small bit of the virtual world. It's a very compelling kind of experience for that reason.

Eric Haines:

Anyway. So let's see what else. I don't know, just yeah, mapping in general. I mean, just seeing things... I was thinking about it today. I was walking around, and one of my things I do is I try to a walk every day. And so my summer project last summer was I'm going to go and map, or I'm going to help confirm the map of all the little free libraries and micro pantry food fridges and stuff in the area where I live. Somerville is the most densely populated city in all of New England, denser than Boston. And it's mostly because of a shortage of green space frankly. But it's next to Cambridge, it's next to Boston, and so on.

Eric Haines:

And so this area there's just a high density of things going on. So in the area there's 200 little free libraries. Well, why don't I map them and why don't I stick a little QR code in the window of each one so that anybody can just take their phone out and go, "Okay, show me the map." And oh, look, there's three others that are two blocks away that I never knew about. And so it's just this very simple thing where it's just me and a bunch of volunteers basically reporting where locations are, and we put it into a map and now suddenly people can go find stuff, find where they can donate their extra canned goods to help someone else out, donate their books and let someone else enjoy them and so on.

Eric Haines:

And it's a very minor thing in a certain kind of way. It's very low level tech in a certain way, but it's not something we could have done a decade ago. There wasn't really the ability to do Google Maps and edit your own map and share that map with people and QR codes. That wasn't really a thing that you could just get a free QR code. I mean, I think there's this sort of things that are sort of subtly happening that we don't... For me, the metaverse the ultimate is the holodeck, right, as we all think, "Oh, Star Trek holodeck. We're all going to have these adventures and some crazy space or something."

Eric Haines:

And it's, well, yeah, you could do that, but there's also just sort of this percolating data is everywhere kind of thing. It's just hiding behind the surface. It's just you have to pick up your phone and kind of look for that data. But it's right there. In some cases it's very close. Another wonderful example is Google Translate. I mean, I hope that everyone knows that if put the Google Translate app on your phone and you hold it up to a sign that's in a foreign language, it'll translate it for you in about the same font, about the same size, and everything. And you just hold it there and you can read that sign. And that's just miraculous to me. I mean, that's basically the sort of unseen information that sort of virtual impinges on the real, and it's so cool.

Patrick Cozzi:

One thing we're hoping to cover today was thinking about the cumulative work of graphics, of the SIGGRAPH community, of folks in games, movies, manufacturing, and how their work has really created the foundations for the metaverse, what it is today and what it can be in the future. And you've already touched on so many things. You talked about kind of ray tracing and its evolution, and then all the advancements here, and digital twins for construction and for mapping, and then talking about in that game environment, users being able to build and manipulate objects, really lowering the barrier to entry for creators. Was there any other kind of cumulative work that you wanted to talk about?

Eric Haines:

Well, certainly graphics in general, for sure of course. It's just we've gone from very simple scenes I mean, things like Unreal's nanite technology is just incredible. The huge amounts of detail that you can get. Yeah. And also, yeah, the compute power is not to be denied. I mean, I work for NVIDIA now. And so the bragging rights is that 7 out of the 10 fastest supercomputers have GPUs as their bases. Well, GPUs wouldn't have been around if there wasn't a consumer market that was sort of in the mid '90s with 3D effects and NVIDIA and AMD or ATI at the time coming out with consumer level graphics cards that brought the prices way down.

Eric Haines:

So it's interesting how the consumer markets actually made it possible for all the other cool things to happen. We're doing digital twins of the entire world's climate so we can kind of see the effects of climate change and so on. But it's due to the sort of falling prices of games and so on of consumer uses. So yeah. So just seeing that, that's sort of a fascinating thing. Yeah. And like I say, my specialty is ray tracing. I've done it for lots and lots of years, And yeah, and I've tried to encourage community by like the Ray Tracing News, which is one thing that Andrew Glassner, and shout out to him, he founded actually as a paper thing. And then I said, "This could go onto... this could just be an email or a Usenet thing."

Eric Haines:

And anyway, that ability to just share information now it's just crazy great. I mean also just seeing things like... One thing I did get to help do was help found the Journal of Graphics Tools, which led to the Journal of Computer Graphics Techniques, which are absolutely free journals basically... or the second one, JCGT. It's all online. It's free. I've been shepherding a paper today, which we sent off the reviews to the authors. We expect the authors to get back to us in a few weeks. And after it goes through copy editing, boom, it's available.

Eric Haines:

And so you can still have a high quality kind of a thing where you have a high quality journal, but you don't have to have the six month lag time. And you also don't have to have the, oh, I just threw it onto... I threw my article onto some free thing, which I think there's still an editorial function in journals, for example. It doesn’t necessarily have to be printed. But that editorial review, that peer review is really important. I think for trying to sort of separate out the good information from the bad and also just to help the author to refine work and make it worth people's time. I mean, there's only so much time you can... You can only have so much time to read articles. So you want to sort of go to the places that you feel are higher quality.

Eric Haines:

And so it's just exciting to see that that evolution has occurred where... Well, we really don't need the print. We really don't need the big publisher with lots of paper and a printing press off in Pennsylvania or whatever. We can do it for essentially free. It's Patreon at worst. You just need really a few bucks for copyediting and that's it. I mean, this is exciting is just how much a single individual can make a huge difference nowadays, as opposed to it used to be just hundreds of people in a company to do something. Now you're seeing it all the time where one person comes up with a great little thing and changes the world.

Patrick Cozzi:

Yeah. And I'm a big fan of JCGT. And I love the philosophy of doing very pragmatic work and having it so available. And I agree with you that one person can really make a big impact. So when you look at the progress of the field, do you think that the second derivative is also increasing, that we're making more progress and making it faster?

Eric Haines:

Yeah. I don't know. It's interesting. I went to a talk, virtual, with MIT CSAIL a week or two ago. And it was pretty interesting. The guy, his name's eluding me. But basically he sort of tracks algorithms. He and his group are trying to sort of look at all the algorithms and look at algorithmic progress and about we might go from an N square... I won't get into the technical details, but from a difficult, a not very good algorithm, to a better algorithm, to a better algorithm. And he was kind of maintaining that's kind of running out of steam on some algorithms, like a lot of sort of major kinds of algorithmic problems, classic problems that, well, we've kind of solved those big ones.

Eric Haines:

And on one level I was like, "Yeah, that's true." There's sort of the theoretical minimum that you can get to. But on the other hand there's always new hardware. There's always new ways of data, what's important. I mean, for example, I think it sums up as, there's always a bottleneck and that bottleneck changes. It used to be back in the '80s we would do instruction counts. We'd go, "Oh, look we want to multiply three times five plus six." Okay. Well, that's a multiply and an add. Okay. That's two instructions. That's really important. Okay. Well that became irrelevant 20 years later where we basically said, no, what's more important is how memory is. Touching memory is slow. The memory is sort of this big pool of stuff you can use, but it's sometimes slow to get that memory. So that's the bottleneck.

Eric Haines:

But then as you think about things like the metaverse or trying internet of things and so on, you go, well, maybe it's the bandwidth or maybe it's latency. It's the round trip. It's I get an information in, my headset responds, and then I respond to that information. There's some latency, some lag there. Is that the important thing and how do we cut that lag down? And so maybe 5G will actually change that kind of bottleneck because 5G is supposed to be much, much faster. Right? And so it'll be really interesting. I'm excited actually to see 5G hopefully really permeate and really become just the norm. I'm hoping that people will go through at least one more upgrade cycle on their cell phones.

Eric Haines:

The cell phone market is kind of slowing down. And I think that's part of the reason we're interested in the metaverse is because commercially it's, ooh, what's the next hot thing. But just seeing how just amazing the phone has become... But I think there's still even more there. So yeah, there's sort of this, oh, in theory we've slowed down, but in a lot of ways we've really sped up as far as going, "Oh, look we can just make these processes cheaper." We're literally putting... oh, I forget what they call it. I think it's the... I want to try to get the term right. Let's see. I'm going to look at it.

Eric Haines:

Yes, it's the “super pod” is the marketing term. There's this thing where they basically hook together a whole bunch of these sort of GPU based processors in a huge system. You get a terabyte of memory. You get petaflops of processing and so on. And that's the kind of thing that we look at now and we go, "Oh, boy that thing's the price of... that's millions of dollars." But you go, well, again, look at the Pixel Machine 35 years ago, that was $150,000. My cell phone is way more powerful than the Pixel Machine is now.

Eric Haines:

So I can't keep promising that Moore’s Law is going to continue and so on, but just there's so much interesting stuff happening in this area that... I mean, that's what's great about the field, right? Every five years, there's a new technology and you go, "Oh, what's that mean?" Anyway.

Patrick Cozzi:

I think it would've been very cool to be around in the '70s when the field was really unfolding, but I'm really happy to be in the field today because we have so much compute available, so much work that's already been done. I think it's just so exciting. So previously you mentioned that you're working on ray tracing and that you're working at NVIDIA. Do you want to tell us more about what you're up to at NVIDIA?

Eric Haines:

Yeah. A few things. Yeah. Let's see, I mean, one thing is this Omniverse project which you guys have already had a whole session on. So I won't go into it too deeply except to say Omniverse – what Omniverse means to me is it's basically a way to pull in data from different sources and be able to do cool things with it like simulations and so on. I'm more focused on the rendering side of things and really I'm kind of a very glorified quality assurance person right now. I'll try out the key framer or I'll try out this various rendering things and I'll be able to give hopefully very informed bug reports about, hey, this is wrong and I think it's because your normal maps are maybe reversed and you're not doing the proper blah, blah, blah, or whatever that...

Eric Haines:

So I'm not programming so much per se, but I'm trying to look carefully that we're getting it all right. And so really lately my interest has been a kind of obscure thing in a way. It's something that we all think we know about, but I realized I've been in the field, I don't know, 40 odd years or something. And I'm like, "Yeah, no, I really don't know that area that well," which is describing light. We talk about, oh, physically based light. We talk about, well, we have a light source and it's giving off photons. And of course to you and me photons means visible light, but photons can mean anything. They can mean microwave or X-rays or whatever. And they bounce off some surface. And so we know a lot about surface material kinds of things where we go, oh, it's metallic and it's this shiny, this kind of glossiness, it has this kind of response and so on.

Eric Haines:

And we know about lights. I mean, actually measuring lights has been around for 120 plus years of how many lumens or candelas or whatever. But it's not something we pay a heck of a lot of attention to in most packages. It's kind of interesting. We just kind of go, "Oh, the lights are dim. Well, make them brighter, turn the knob." And so I know NVIDIA is definitely working on it with other people. They've got sort of a serious group that's working on getting that just right for simulation, because you want to be able to take real world data and bring it into your simulated environment and have it respond properly to the right lighting. And you can say, "Well, this sun's a 100,000 lux, and it'll actually work properly inside your application."

Eric Haines:

And it sounds sort of trivial like, "Oh, he just set the light to be properly correct." But I've been doing sort of a survey of the field, and there's not that many apps that are really sort of generally doing that kind of thing or doing it well, doing it nicely. So anyway, that's my own personal sort of rabbit hole. But it's been a very interesting rabbit hole. I mean, this weekend I may be going into literally a rabbit hole. I may be going into my closet and putting a blanket over my head and holding up a 60 watt whatever that is. I forget how many lumens light bulb that is. And putting a little gray card on the floor and getting my fancy $8 app on my iPhone so I can set the exposure just right. And seeing if I can run the numbers and get the right answer, like is this really working as I should expect it to?

Eric Haines:

And it all sounds minor, but I mean, I've been talking with other people like a games company, for example there. They're doing the same kinds of things. They're going out and going, "Yeah, we're just buying some cheap equipment." You can get your lux meter for 20 bucks off of Amazon. And just trying to cobble something together in the garage to go, "Well, is our game kind of responding the right way to light, and what are we missing here if we don't do it right?"

Eric Haines:

So it sounds like a very silly niche in a way, but it's one where I realized I don't really know much about that area. So I've spent the past few months kind of in my spare time kind of reading about that. And it's been fun to read about. But it's just one element that it's, oh, it sounds minor, but getting it right makes it just much easier for us to interchange data.

Eric Haines:

Speaking English, I want to talk about USD a little bit because that's the other thing I've actually been working a lot on in the past two years or so, is just trying to get USD... or trying to understand USD for one, just what its materials and what its lights and so on and camera model is. And just testing it. Like you mentioned, I got seriously addicted to Minecraft back in 2012 or whatever, and I went through the 12 step program. I'm clean now. I don't have my Minecraft addiction anymore. But what did come out of that was I wrote this little program, free program, called Mineways, which just lets you export data from Minecraft to a common file format.

Eric Haines:

And I started with this ancient one called Wavefront OBJ from the '80s, but it's still used today. And so it's sort of with USD coming up, I'm like, "Ooh, I should export to USD because it's a much more full featured format." I mean it's actually kind of pitiful that one of our formats that we're using nowadays and I still see it used all the time, just OBJ from the '80s. And so I'm very happy about seeing you and others using glTF and pushing glTF, because it's like, oh, yeah, we need a new interactive format that's just better. And so it's been great seeing glTF really come into its own. And I consider glTF kind of in some ways the big brother to Pixar's USD, and I think USD can learn things from glTF as far as just some of the definitional stuff that you guys are working on that you're a few steps ahead on a few things. And I've heard that in your podcast where USD people are like, "Yeah, yeah. We like glTF and we like their ideas and so on." So that's great to see.

Eric Haines:

Anyway. But for me anyway, fidelity has been a really important thing, is just... Omniverse versus supposed to be well, we're taking in data from all kinds of different apps. Well, if that common material format if I'm pulling it in from one app and it looks different than what I'm pulling in from another app, and they say that they want to be the same but they look different, it's broken as far as I'm concerned. So I've been using that, my little Mineways exporter to test USD. So it's the geometry you don't care about. It's all just blocks. Minecraft is all blocks and cut out objects and that's it.

Eric Haines:

But it can get very elaborate. The hobbyists out there have gone nuts. And you can get very elaborate metallic with normal maps and reflectance and all this kind of crazy stuff attached to these models. So I've been doing a lot of just putting out that data, seeing where I've gone wrong, fixing it, whatever. And also it's just been great for testing both Omniverse and a bunch of apps, a bunch of popular digital content creation apps out there and just finding out, "Ooh this isn't so good. We got to get a little more standardization here."

Eric Haines:

One thing I wanted to mention was many years ago I worked on VRML a bit, which is sort of a precursor, sort of yet another file format in a way, but it was supposed to be sort of the harbinger of, oh, we're going to use this for VR and this is going to be the metaverse and all the rest. Right? And it never really caught on. And I think for a few reasons. VRML 1 was really solid. VRML 2 was solid, but you know what? It was kind of a write-only language because they added all these sensors and simulation kinds of things like time sensors and proximity sensors and touch sensors and stuff. And it soon became unreadable by anybody other than the person producing the file.

Eric Haines:

And so that's I think one of the dangers that you guys face and that USD faces and so on, is it's easy to extend things but you kind of have to make sure there's buy-in and that it doesn't evolve so quickly, like something doesn't get standardized that people are like, "Yeah, that's part of the standard, but I don't implement that." And that's a problem. If it's not getting implemented by the majority of users or by the majority of implementers then you're really in trouble. Anyway, that's my cautionary tale because VRML pretty much died. I mean, you still see it used for 3D printing actually, oddly enough, but not much else as far as I know. But back in the day we were like, "Oh, it's going to be great." And it was. It was actually really a beautiful effort, but for various reasons, wasn't quite there.

Patrick Cozzi:

A lot of good stuff. So I think the USD/glTF admiration, I think is mutual. I think there's a lot that glTF can learn from USD. And I hope that both evolve in the future in a way that maybe there's some transcoding in both directions. The challenges that you mentioned around consistency for physically based rendering materials or any materials, that's a tough problem. I know Khronos looked at it with COLLADA kind of way back in the day. We certainly looked at it with glTF, and they may still be looking at it. I feel good knowing that minds like yours are looking at it. I can't wait to see kind of what you come up with.

Patrick Cozzi:

And then finally that kind of lesson learned on extending VRML from 1.0 to 2.0, certainly with glTF we were very careful trying to do a minimal but useful set for efficient 3D runtime asset, allow folks to extend it, but try to extend at a reasonable pace so that you can get it adopted and then still have a robust interoperability. So that's a tight rope that is a challenge to walk.

Eric Haines:

Yep. For sure. I mean, yeah but a necessary element. I mean, it is great to see that we've gotten a lot closer than we used to be kind of thing as far as being able to interop. We always want to do one more thing. "Oh, I just want to add one more feature," and so on. But I appreciate the pace is, really trying to think that through carefully, because yeah, once you've sort of added some crazy thing it's hard to pull it back not be able to do it anymore. I mean the Three.js guys can do that, because that's just a hobby project and that's a project that I love, is just this thing, oh, Three.js. It's a great way to just learn about graphics. It sort of hides all this sort of WebGL complication. You can just sort of say, "I want a sphere and I want a camera and I want to put a light over here." And in a few minutes you can get something on the screen in your web browser, you can send that link to somebody and share it and so on.

Eric Haines:

But those guys are pretty wild as far as standards. It's the opposite. They'll just go, "I don't like that feature. I'm throwing it away. And now we're doing some other feature." Like I say, it's a great area to learn in and so on and play, but it's a little bit tricky if you're trying to have any kind of continuity of software and being able to keep your software running on that platform. You pretty much have to lock into place and say, "Oh, I can't move from here. I am stuck." But anyway.

Patrick Cozzi:

And you reminded me of an interesting story from a Khronos BOF at SIGGRAPH. And you were there. It was the BOF when Amanda Watson from Oculus came up and talked about their use of glTF. And after that Ton from Blender, he and I were chatting, and he gave me perhaps the best advice I ever got for glTF, which was keep glTF simple, get it to 1.0, have a lot of folks adopt it. And then you can add to it slowly. He's like, "Don't put everything in right away."

Eric Haines:

Thankfully you got the message.

Patrick Cozzi:

Man. So very influential. So Eric, the next question I've been waiting to ask you. So look, I want to ask you how folks can get into the field, into graphics, into the metaverse. And look, I really believe that there's no one else better in the world to ask than you. And certainly when I think of myself, I mean, you helped me get into the field. And I did a little research before this podcast and I went back into my old Hotmail email and I found the email when we first met. It was January 2010. And I sent you an email and you didn't know me. And I said, "Eric, love your books. I did this masters thesis, this cool 3D globe stuff. I'm going to write this book. Can you give me some advice? Will you be at i3D?"

Patrick Cozzi:

And that same day, you didn't know me, you replied and you said, "I won't be at i3D. Here's some advice and here's my phone number." Wow. I mean, look how welcoming the graphics community was, and that you really helped me a lot over the years in many facets, not just the book. So I would love for our listeners, whether they're in college, maybe they're in high school, or maybe they're in another field or maybe they're in graphics but they want to do something else in the metaverse, I mean, what advice do you have for us?

Eric Haines:

Yeah. There's plenty of ways to get involved. I mean, that's the beauty of the field right now is just the fact that there's this whole internet out there that you can do things like look at Babylon.js or Three.js. And you can learn and there's bunches of tutorials out there. I mean, I taught a MOOC about this what, eight or nine years ago. So that one's old. So don't use that. I mean there's some good material there. But like I say, Three.js has kind of moved on and done other stuff.

Eric Haines:

But honestly, I mean, just looking at those kinds of packages, if you're interested in programming or even if you're just interested in like, "I just want to play with some objects and put some stuff on the screen and send my grandparents a link and they can look at this cool thing that I did," that's all to the good. It's very easy to sort of just jump into some of those packages and it's you just have to kind of follow the steps carefully like, "Okay, wow. I just use my text editor and I changed this sign and what happened to the light? Oh, okay. It changed that. Cool. Okay. What if I change the shininess of that sphere? Oh, okay. That's what that does." And by just experiment, you can sort of quickly get some really good solid honestly graphics knowledge from just your own experimentation, if you want to do it that way.

Eric Haines:

But there's also... projects get involved in. I mean, there's people that are writing mods. So if you're more artistically inclined, you can kind of just get involved in the modding scene of okay, well I want come with some textures or want to do... I'll make a model for this thing in Blender, or I want to do some animation or whatever. Again, it's sort of one of those you choose... It's choose your own adventure. You go, "Oh, wait, I want to know about animation." Well, oh, okay, well, Blender. Yeah. That's free. And you can find a ton of stuff on YouTube and all over the place about how to do this, that, and the other in Blender. So that's a great venue.

Eric Haines:

Traditional ways obviously. Go to school. Having a solid education, some kind of either in the arts or computer science or a mix of both really is... The double threat kind of thing is to have really both, if you can know something about modeling and art and how perspective works and all that kind of stuff. That has a direct connection to graphics for sure. So yeah, I think there's a lot of interesting ways there. What other things?

Eric Haines:

Also yeah, papers are accessible. I mean some of these papers, yeah they're probably going to go over your head. There's good blogs out there. You can look for those. I'll put a plug in for realtimerendering.com. There's a page, which I call the portal page, which I have left at that because that was the name people used 20 years ago. And I've always kind of cracked me up to hear this portal page. But that's just sort of a page of resources and it has a bunch of blogs listed that I think are pretty good. And so that's a great resource to just go and dig into that page and look at stuff there.

Eric Haines:

What else? Yeah. Like I say, read articles, or if you see someone that hey, when you're doing something cool, write them. Don't feel shy. I mean, that's something that really impressed me about you to be honest, Patrick, is that you weren't shy. You were like, "Oh, there's this guy. I'd like to meet him. Well, I think I'll go try to write him and see what happens." And that's one thing I think you've heard me say it before, but is if you want to contact someone on LinkedIn, you want to make connection on LinkedIn, don't just hit the button that says make a connection on LinkedIn because most people either they'll just accept it like, "Yeah, I don't care." Or they'll go, "I don't know who you are," and they'll refuse it.

Eric Haines:

But if you actually write them and say, "Well, I read your article. It was really interesting, but I didn't really understand this part," or, "Oh, I wanted to talk to you about that," or whatever. That's something I'll respond to. I mean, that's something that if you see, "Oh, wow, this person's putting in more than the two seconds that it took them to click that connect on LinkedIn button," that means something.

Eric Haines:

People like to know that their work is getting used. If you've written a book or an article or whatever, and someone writes to you and says, "Oh, I have a question about this," I'll try to find the time to respond because you're the one in a thousand who's actually making the effort, interested in the topic. And I want to encourage that. So that's where I tend to go.

Patrick Cozzi:

That's such good advice. And once in a while people do email Kevin or I about the 3D Engine Design for Virtual Globes book 11 years later. And we really do make an attempt to give sometimes a useful response. So I think that's great advice. And then you encouraging folks at the bar to enter to get started with a Three.js or Babylon is so low is great. Right? You can jump in and experiment. And then the fact that you can contribute and build either on the technical engineering side or on the art side. Right? I think there's many ways to play. So that's awesome. So Eric, we've talked about so much. Is there anything we didn't talk about that you want to?

Eric Haines:

Oh, I could blab on this stuff for hours, but no, not particularly. I mean, like I say, I think it's just interesting to sort of see... Well, you're in the sort of GIS field, and it's just fascinating to sort of see, like I say, that sort of permeability or something between sort of virtual and the real world. I mean, unfortunately we're seeing it in things like Ukraine and so on where open source intelligence, the Bellingcat people and so on, they're trying to figure out things.

Eric Haines:

But I think there's just open source mapping or I just... It's fascinating. And I think we'll kind of go in 20 years, like, "Wow. They didn't wear cameras back then that kind of just mapped their whole area and detected potholes or detected... That's just the norm now." You have these sensors that are picking up data. Right now we're sort of individually smart sensors. For example, in Somerville, we have this 311 app, and we are the first ones in the state and now a bunch of places have it, which is you basically have an app where you're standing somewhere and you're like, "There's a pothole right here." You bring it up. You say, "There's a pothole right here." And it'll, obviously because of GPS, it'll know where that is and all that. And within a week it'll get fixed, which is really miraculous if you think about it, because that's something you really had to have a big effort to tell where the pothole was 10 years ago.

Eric Haines:

And I think it's going to be the same as we go. It's like there's sort of, to me, a little bit scary nanny state kind of thing where we're putting cameras on all the corners, but I think that can happen in different ways. And we're already sort of seeing that with people with their cell phones explicitly photographing things. But I think you can also have sort of an implicit, like you are sensing something about you and you're willing to share that data. And I think that's just huge. That will just continue. And we're going to just see a lot of interesting things going on there.

Eric Haines:

Anyway, that's my only other kind of obvious thing. Many people have said, but yeah, you just see it coming. I think there's just a lot of interesting ideas coming out about that kind of stuff.

Patrick Cozzi:

Yeah. And it's accumulation of so much technology, right? It's so many different types of sensors. They're widely available. There's more bandwidth. There's more compute, the appetite for digital 3D content. Yeah. It's interesting.

Eric Haines:

Yeah. Yeah. Well, I mean, even in the next year, I'm looking forward to, I guess, Snap with Snapchat. They're having some spectacles come out for AR. And Apple is rumored to have something next year, maybe next spring or something. So we'll see. But yeah, no, it's just... I don't know what it'll be that triggers things. I mean, as much as I love the Oculus Quest 2, it's kind of a biggie heavy, bulky thing. It's not what we all dream of, which is the lightweight glasses that we can maybe tap and do something with, but there's other things going on. I mean, the technology of just being able to talk to your sort of smart speakers, Alexa. I ask it for the weather. I put a timer on. Maybe that's not it, but walking around, I'd be like, okay, wait, I want to talk and sort of immediately get some kind of response. Yeah. I can imagine that happening on a different form factor.

Eric Haines:

But anyway, it's exciting to think about. But not my area of expertise. Hopefully, the people listening to this are the guys who are... You're all doing this cool stuff and making it happen. So go for it. Great.

Patrick Cozzi:

Eric, we like to wrap up the show with a shout out. Is there any person or organization you want to give a shout out to?

Eric Haines:

I do. I was thinking about this and it's kind of a silly one in a way, because it's sort of very old school but I think it's really critical, which is archive.org, which is just kind of where all the old dead web pages can be found because people will have a cool blog or cool articles or whatever, but they're not published or they're not formally put into some archive. And so archive.org is great. I mean, Wikipedia depends on it really. If you go on and click on a footnote, they'll often just go right to the archive.org thing because they know that'll be there as opposed to some web link that the webmaster rearranges the website. And even if the website's still there, the link's dead. So they do good things. I just contributed some money to them. So I recommend you all give them a few bucks if you can because it's just a huge deal.

Patrick Cozzi:

Excellent shout out. Eric, thank you so much for joining us today, thanks everyone for listening. We really appreciate it. If you have any comments, please leave a note, give it thumbs up, give it the thumbs down. Let us know what you think. And please subscribe. Thanks everybody.