A History in Computer Graphics with Kurt Akeley

Announcer:

Today on Building the Open Metaverse.

Kurt Akeley:

The insight that I evolved over time and the change I made at Silicon Graphics was to identify the problems we're solving, figure out a useful interface, design systems, and then implement parts that were needed for the systems. There's nothing pure about it. There's all kinds of cycles, it's a complicated process; But the cart behind the horse instead of out front, substantially, was my story and SGI’s story as we moved along.

Announcer:

Welcome to Building the Open Metaverse, where technology experts discuss how the community is building the Open Metaverse together, hosted by Patrick Cozzi from Cesium and Marc Petit from Epic Games.

Marc Petit:

Hello, I am Marc Petit from Epic Games, and my co-host is Patrick Cozzi from Cesium.

Hey, Patrick, how are you today?

Patrick Cozzi:

Hey, Marc. I'm doing great. I'm very excited for today's guest. You may not know, but before we met and even before Cesium, I spent about eight years specializing in OpenGL and using it every day, so we have a very special guest.

Marc Petit:

Today, this is your show, Patrick, because our guest, and we're super happy to welcome Kurt Akeley, a true pioneer in computer graphics who led the devising of OpenGL, amongst other things, and we'll talk about his contribution to computer graphics.

Kurt, we're so happy to have you with us. Welcome to the show.

Kurt Akeley:

Thank you very much. It's great to be here.

Patrick Cozzi:

Kurt, thank you so much for joining us. We love to kick off every show by asking our guests about their journey to the metaverse or their background.

For you, you've done so much amazing work at Stanford, working with Jim Clark and founding SGI. Tell us your story.

Kurt Akeley:

Well, as far back as I can remember, I have been fascinated by understanding how things work and then wanting to develop my ability to affect how things work. I had to design things and build things and fix things, and the tools behind me are a clue that hasn't changed.

By the time I was in high school, I was playing with TTL circuits and building circuit boards, and in college, I ended up doing a pretty substantial project with a guy, the head of the EE department, who had been a vice president at Xerox previously, and together, we built one of the world's first laser printers, repurposed a telecopier, I think Xerox called it. I did all the circuitry and got all the mechanics of the xerography working, all this stuff.

By the time I finished my undergraduate degree, I had a pretty good idea of how to do pretty significant system implementation. Found out I could get a PhD in engineering. I didn't know that until my undergraduate was almost done. A philosophy degree in engineering. Who knew?

I ended up going to Stanford and joining the PhD program there. I was full of energy to find a professor to work with. There were 50 or so EE professors, and I couldn't find anybody who was building stuff, and I was just itching to build things.

During a seminar course early on in that first year, the course was being organized by a guy named Jim Clark; he got a speaker every week, and one week the speaker didn't show. So Jim got up and did a song and dance and talked about the amazing things he had done, in terms of virtual reality, really, way long ago, and indicated he was looking for students interested in doing something interesting. I just about jumped out of my seat.

So, I made an appointment, visited him in his office, and showed him the 600-page document that I had written and printed on that laser printer that was able to describe the whole thing and how it worked. He hired me as a research assistant for the next year as a result of that. 

I spent the first few weeks of the following summer, he was nowhere to be found. He was busy developing this geometry engine, so I just went to the math science library and read Newman and Sproull and tried to learn something about computer graphics beyond the simple glyph composition that the laser printer had required. I had built that.

Marc Petit:

He was already working on the geometry engine at the time?

Kurt Akeley:

He was.

Marc Petit:

Can you tell people what the geometry engine was?

Kurt Akeley:

Sure, just a critical part of what makes interactive computer graphics interesting is that you can define the objects in a coordinate system that's convenient for their definition and then transform them into a coordinate system that is appropriate for rendering a display.

Mathematically, that's straightforward, but at the time, doing it fast was quite challenging. Jim took on that challenge. He knew that there was amazing work in integrated circuit development in a way that was accessible to mere mortals. He knew quite a bit about that transformation process, so we thought he would try to take those two bodies of knowledge and use them together to invent a little computer circuit that would do those sorts of transformations; he called that the geometry engine. He actually took 12 of them all together to do the job. They were geometry engines that collectively did that interactive transformation.

Marc Petit:

The ancestor of the GPU, the mother of all GPUs.

Kurt Akeley:

It probably is, right?

I worked for him that summer and proposed a backend for this geometry engine to do some rasterization. Then I bumped into him, and he asked if I would be interested in starting a company with him. Crazy guy that I was, it turned out I was interested in doing that, which was a disappointment to my parents and the people that had gotten me into that Stanford PhD program.

I took a leave of absence, and he and I, and about eight other people, all collectively founded Silicon Graphics, and we were off to the races. I did get the chance to finish that degree, actually, but that's a story, maybe, for later.

Marc Petit:

That was a few years later, right?

Kurt Akeley:

Yeah.

Marc Petit:

Well, also, a word about SGI, I studied my programming career on the Personal IRIS 4D/20.

Kurt Akeley: 

Yep.

Marc Petit:

And I was supporting ILM at the time they were working on Jurassic Park, I was at Soft Imagine.

Kurt Akeley:

Excellent.

Marc Petit:

SGI, at the time, in the early nineties, really transformed Hollywood.

There was even a cameo of the SGI file system in Jurassic Park.

Kurt Akeley:

In Jurassic Park, I saw that.

Marc Petit:

That was crazy. The impact that this company had on graphics in Hollywood through all of that 3D software and software like Flame has been nothing short of a... I mean, we forget.

Kurt Akeley:

Yeah.

Marc Petit:

Maybe we get to talk about the transition from SGI to Nvidia and what happened there, but maybe not.

Kurt Akeley:

I'm happy to. I think I probably will, even if you don't ask.

Marc Petit:

SGI was such a huge innovation leader.

We heard about Jim Clark in another episode with Ping Fu, who was overseeing his thesis on Netscape; he was involved in the Netscape endeavor with Marc Andreessen.

Kurt Akeley:

I would actually argue that Nvidia is what Silicon Graphics aimed to be. In retrospect, SGI, Silicon Graphics, the SGI would be better standing for systems graphics. I would argue that collectively and during that period of time, we systematized how to do interactive computer graphics on computing systems and solved a bunch of interesting problems and a whole bunch of things.

But we never were successful at producing silicon to do interactive graphics, nor did we ever build a successful sort of high-performance computing combined with interactive graphics conglomerate. That was the ultimate dream. And Nvidia has done both of those things.

To me, Nvidia is Silicon Graphics as it evolved, and indeed, since I work in the valley, I sort of think of us all as being employees of this valley. The companies have functional reasons, but in a sense, SGI evolved up and became Nvidia, and the success that Jim envisioned has actually happened.

Actually, a lot of the same people are there doing it. That's a short for how I feel about Nvidia. It is what Silicon Graphics aimed to be and SGI did a lot of really useful system-level stuff, but we never got all the way to that endgame.

Marc Petit:

Throughout your history and your career, you've been involved with all aspects of computer architecture. You've done hardware design, systems software, you've written CAD tools, you worked on display technology and programming languages–we'll come back on Nvidia and CG.

As you look back, are there any big takeaways? Do you have any favorite projects? Do you even have any regrets about all of those things?

Kurt Akeley:

I think I could describe my first 10 to 15 years at Silicon Graphics as this pathway from working on very, very low-level things. Thinking I'm going to be designing integrated circuits and studying for that at Stanford and then actually viewing TTL board-level designs and then ultimately moving up through that to ASIC design, gate arrays, not field programmable gate arrays, but the original gate array stuff and then ultimately APIs for the systems.

I think the insight was that we were doing things backwards when we started at SGI. We were doing things in the order I just specified: figure out what's a geometry engine, then figure out how to put geometry engines together, that system and what other pieces we needed, and then figure out how that system wanted to be interacted with by people who are using it. And then try to find what problems could be solved with it.

I think the insight that I evolved over time and the change I made at Silicon Graphics was, as much as possible, to reverse that. Identify the problems we're solving, figure out a useful interface, design systems, and then implement parts that were needed for those systems.

You have experience with this; there’s nothing pure about this. There are all kinds of cycles. It's a complicated process, but just in the aggregate, trying to get the cart behind the horse instead of out front was sort of my story and SGIs story as we moved along. I think that had a lot to do with how we got and where we were able to get, which was, as you observed, a pretty significant place by the mid-1990s.

Getting things in the right order is one, and then one of those pieces of that order is what I call API or interface. The other insight about all this was, ultimately, that it isn't just an interface; it needs to be thought about in a very special way. And there are names for those ways, we can get to that.

Patrick Cozzi:

Kurt, that's a nice segue because I've been dying, of course, to ask you about OpenGL and your thoughts there on APIs versus architectures.

Before we jump in, though, for those of us who are maybe not listening but watching on YouTube, you have one of the coolest backgrounds I've ever seen on a Zoom call. Would you be open to sharing that with the group?

Kurt Akeley:

Sure.

We built this house, finished last year, and it has this room in it, which is my workshop room. I'm an engineer, and I love tools. I love using tools and understanding how to use tools and having tools, all these things. What you're seeing in the background is just one of many peg boards and collections of tools. There's far more out in the garage, but it's cold out in the garage. These are the ones that are useful for doing indoor work that doesn't generate too much sawdust and the heavy equipment and the like are out in the garage, which is occupied mostly by my stuff and not by vehicles.

Marc Petit:

So what is the latest thing you've built?

Kurt Akeley:

The latest, I guess, was putting trim around the shelves that I put up in the master bedroom. For this house, a lot of things that my wife identifies, things that are needed, the arbor's out in the gardens that are all made out of nice redwood and finished much better than usual outdoor things are because I just like having the time and ability to do things slowly and well.

It's a delightful aspect of being retired, although I would say even in my professional career, to a substantial extent, I took my time and did things slowly and well.

Patrick Cozzi:

Kurt, let's jump into OpenGL.

In 1992, OpenGL allowed programmers to do legit cross-platform 3D, which created a tremendous impact. Before we jump into that impact, though, let's talk about a topic that's near and dear to your heart, which is computer architecture. What is it, what isn't it, and why is OpenGL an architecture?

Kurt Akeley:

Okay, well, let me start with the story of how we got to OpenGL. I'll try not to get too long, and then you may have to remind me to get back to answer the question because I could get lost.

Jim Clark had really good instincts; he and a small team developed this IRIS GL, which was basically an interface to the computer graphic systems that we were building in those early years at SGI. What became clear over time as we licensed it to IBM to use and as we tried to port it from one of our machines to another was that it really wasn't a very well-defined interface, that we found ourselves doing experiments to try to figure out, gee, if you do this and that, what happens?

We would do the experiments on the old machine and then try to make the new machine work like that.

That's not a very disciplined or thoughtful way to go about things. Then, again, you license it to someone else, and they're trying to make it, and it just gets to be a mess. From an implementation standpoint, it was an inadequate specification, and from a porting standpoint for people using IRIS GL, it was difficult to write a program that would easily run on a different platform than the one that it was developed on. 

It turns out these are classic problems. These problems were identified and addressed and substantially understood and solved by Fred Brooks and the team that designed, conceived, and implemented the IBM 360 back in the early 1960s. I think it was introduced at the World's Fair in New York in 1964. I happened to be there... Now, I didn't have any idea what was going on. I was six years old.

What we really needed for our graphic systems, what the industry needed, was an interface that had these kinds of disciplines about it that Fred had figured out a name for such an interface as an architecture.

IBM 360, that architecture is an instruction set architecture, but it needn't be an instruction set. In the book that Fred and Gareth wrote years later, they describe the architecture and implementation of a clock, for heaven's sake. I mean, anything can be thought up in this way, and it turns out graphic. So to my embarrassment, because I knew Fred Brooks at the time, I didn't know about this work, and we refigured it all out using different words and different names, which is not a great thing.

But what we were doing is solving the same problem with the same approaches as Fred Brooks had. Now, there are other aspects to that story. I guess the other thing is other people solved these problems as well, and the X Community, the X Window Systems community, was moving ahead with a standard that it evolved from an old graphic standard programmers' hierarchical interface for FIGS. I'm not sure exactly what it stands for; using that in motivation, we're coming up with a thing called the FIGS extension to X or PEX.

I was aware of this at SGI, and we thought that this was kind of a disastrous, perhaps too strong a word, but a substantially less optimal result if the world ended up using that, for a whole bunch of reasons.

One of them, not to be discounted, was that even at that time, we're talking early '90s, the X Window System ran on maybe 1% of the computers that people used. The other 99% were PCs that are whenever else. And yet, this standard was being implemented in a way that was going to be linked only to those X Windows Systems computers, which just looked like, from a standpoint of the overall industry, not a wise move.

We, led by me, but at Silicon Graphics, set out to do a standard, really an architectural standard, that would be better for us, and we thought better for the industry.

A long-ish answer, but that's some of the history of how we found ourselves doing the standard.

Patrick Cozzi:

That's really awesome. Talking about that impact on industry, if we look at it from the perspective of the end user, how would you describe the impact of OpenGL for folks working with interactive content, whether it's CAD or animator visual effects? How did it impact how they worked and their workflows?

Kurt Akeley:

It's more of a business change than a technical change. Most of the things that OpenGL had when it started, all of them were available in IRIS GL and available on the HP interfaces. It's just that those things weren't really available outside that 1% of computers that ran the X interface or X Windows System, and they were expensive.

What happens in architecture like OpenGL is a contract. It's a contract between people who implement some mechanisms and people who use those mechanisms. The contract has a lot of nice properties, and the net result is that it creates a marketplace or wildly accelerates a marketplace that otherwise struggles because it doesn't have the portability, because it doesn't have the reliability, because the interface didn't allow the right kind of cleverness underneath for implementers or it didn't allow portability for reviews.

All kinds of bad things happen when those interfaces aren't good.

By creating that interface, by having that contract, it allows investment to happen, at all levels, money, people's time, it allows multiplicative goodness to happen. The net result is that it started spitting the 3D model of an airplane on the GT.

That was the first time any of us had seen such a thing, right? I mean, sure, some ENS system or flight simulator had done something like that, but literally, nobody saw those that the SGI machines, and an interesting number of people did. But by creating this marketplace with our architectural standards, in some sense, everyone, I mean, your mobile phone does it just fine now, in fact far, far better than any of those expensive things did then. That's the difference. It isn't that it was the first time it ever happened, but it became ubiquitous that these sorts of interactive experiences with computers could happen.

Marc Petit:

You characterize it as a business decision. But at that time, Silicon Graphics was the clear leader in graphics, and you somewhat enabled competition. Was that a hard decision?

Kurt Akeley:

Well, I mean, Jim wasn't happy with me about it. It's a tough call.

I think the trade-off, the way I saw it at the time, was if we stick to proprietary, we're going to be going it alone. Worse, we'll be forced to adopt the things that other people put out there as standards, and that's going to compromise our ability to do what we want to do. You don't get to replay these things. I'm sure some B school person could do a bunch of studies across it, but I was happy at the time that by creating the standard, we were able to move ahead as a company and keep doing the things that we wanted to do.

I didn't join Jim Clark because I was a cutthroat, clever business kind of guy. I wasn't, and I'm not, right? I'm a technologist. It was a relatively easy decision for me to expand the tent and bring people in and have it become a standard.

I think Jim did have a clearer understanding; I think it's quite possible, Marc, that SGI would've maybe lasted a little bit longer in the '90s as a force if we had kept things proprietary. I don't know. But we didn't.

Marc Petit:

I do suspect you were right that history, in hindsight, was on your side.

Actually, you revived a painful memory because when SGI licensed the technology to IBM and created the RS/6000, which was a monster, a non-functional monster, I was on the team at TDI, in charge of porting Explorer to the RS/6000; it would not work.

That's how I found myself at Softiva, actually, because I could not deal.

Kurt Akeley:

Well, I apologize because part of the problem was that it was impossible to port IRIS GL successfully. So there you go, right? I've never had confirmation this way.

Marc Petit:

Let's stay on the topic of open standards because it's one of the things we do on this show when Patrick and I are actively involved in the standardization effort around the metaverse and trying to get the industry to agree on ways that we could represent interactive content, just like HTML was the representation of the hypertext standard.

The same for virtual worlds and working around USD and glTF. I know you've been working with The Khronos Group, and Neil has been on the show a few times; we are partnering with Neil on the Metaverse Standards Forum. This is all the same family.

Do you have advice for us as we set up to try to standardize something even bigger than just the graphics component, which is that metaverse and those fully simulated worlds? How should we go?

Kurt Akeley:

I can reflect a little bit on my experience. I'm not sure if it maps well or not, but one thing is, remember, I was at SGI, and SGI was a dominant force, so we had a real advantage. My experience with how to go about this is a little biased or distorted by that because we could invite people to join us, and they did. Our business didn't matter if the work we had.... I had that advantage, but early on, I really set out to earn the continued trust of all of the people or the people and the companies who were involved.

We did that by being absolutely scrupulous about not taking advantage of our position of strength in the group, in the market, by building a culture of voting and of appropriate behavior of not doing nasty IP kind of things, which the world has learned a whole lot more about the sort of bad things can happen.

SGI, at least, I don't believe did any of that. In fact, we actually indemnified people against patent infringement in the OpenGL license, which is a crazy, crazy thing to do. I can't believe that happened.

Building that trust among the participants and then also arranging things so that the people participating in evolving standards are the right people.

You want to build an organization that attracts really good engineers, so they feel like their ideas will get a good hearing and things will be done; they're a little political.

One of the things that we did, for example, is there was a whole lot of obvious stress about conformance testing. If you're going to have an architecture, you want to make promises in a contractual way, you'd better be testing those promises. Boy, what an opportunity to injure your competition by putting things into tests that are preferential or whatever. It's obviously a minefield.

One of the things we did is we split the working committees for that kind of thing to be separate from the committees that were evolving and defining the standards.

Let's not stir all this together in one pot. You can take little steps like this, organizationally, to keep that trust and keep attracting the kind of people that you want to have involved. I think OpenGL got off to a really good start that way. I'm so pleased with the way the extension things went, for example. We saw the X Window System had an extension mechanism, but it had a nearly fatal flaw. The X extensions were defined against only the core, not against the other extensions. You end up with a center core with a bunch of little pieces stuck on and not an evolving architecture. We really wanted the evolving architecture, so we just made some suitable rules. We'll index the extensions, starting at one, and when you write one, you've got to describe your stuff with the respect to all the ones that have indexes lower than yours.

It gets to be a big problem.

Later on, the spec evolves, and then you sort of radius the playing field. Now you don't have to write against all the earlier. But the point is, we identified what we wanted, which was the evolving standard. Then we looked at what was out there and found the problems. I think there are, last I looked, over 400, I think, OpenGL extensions, and by no means is it perfect.

You can hear plenty of horror stories about issues between extensions, but you wouldn't have 400 extensions if it didn't basically work, right? It would've ground to a halt decades ago, literally decades ago. And it didn't. So like all things, not perfect, but much better.

Marc Petit:

Patrick and I go through the Metaverse Standards Forum experience, and trust is by listening to everybody.

You have to go slow and spend a lot of time together, which is against our nature as entrepreneurs, and we want to get  it done. We are bred to go fast and have milestones and converge quickly.

Here, that trust-building exercise sometimes feels, it feels slow, but I think it's...

Kurt Akeley:

That's right.

Marc Petit:

When you have a group that can actually openly debate the issues, this is where the real progress starts.

Kurt Akeley:

When I started working for the guy building that laser printer as an undergraduate, he hired me for a summer and set me down with all the work that hadn't been done by others. I just worked eight hours a day and read and organized and figured out what everything was and gave everything a name, but he was getting really nervous. I mean, I was two or three weeks into this, and nothing had happened, but then all of a sudden, boom, when it's like, okay, here it is, this is how they all go together, this is what I'm going to do. Then we were moving quickly. It's just taking the time to make the tools, to make the jigs, to figure things out.

Just to be fair, quick prototyping and failing quickly is also really smart.

It isn't just from hardware to software to product to customer, everything has cycles, and there's no pure answer, but boy, putting in the time to think things through.

If I write a paper, the first thing I do is write a glossary; it doesn't matter to me if the glossary is going to end up in the final paper; that’s not the point. The point is I'm going to use vocabulary, I'm going to use words, and they should mean something, and I should know what they mean. Heaven forbid, they shouldn't mean multiple things that are by convenience.

A little bit of rigor about that goes a long way to clarity and on and on.

Patrick Cozzi:

Kurt, I wanted to ask you a bit about levels of abstraction.

If we look at kind of the GPU architecture evolving over the years, then maybe the expectations of the end user in terms of visual quality, visual fidelity, the complexities of the scene, and then also just wanting more and more people to be able to be 3D creators and 3D developers.

When you think about the level of abstraction of OpenGL or maybe a Vulcan or even a game engine today that maybe you code against it or has no codes, visual. I'm sure it's more than one level, but what are the levels of abstractions that should make sense today?

Kurt Akeley:

I want to come to that, but I want to riff a little bit more just on levels of abstraction first. Again, you remind me if I don't get to the end.

It's really, really central when you're defining an abstraction, getting the level right is hugely important. One of the ways to understand that is that, as you've strongly alluded to, there needn't be only one in the system. There's likely a stack of software or hardware or whatever. There are multiple levels of abstraction. The ones that I have the most experience with are the lower ones. Even an architectural abstraction might not be the bottom. Certainly, a microcoded machine has its own microarchitecture and another level of abstraction inside.

I've spent most of my career down at these low levels and not spent a lot of time thinking about or interacting at higher levels of abstraction except to the extent that I need to understand what's needed up there to do the right things down lower.

One thing that is an interesting, I think, fact about the right level of abstraction for interactive graphics sort of toward the bottom is that I think OpenGL hit it pretty well. Again, riffing off what Jim and the team had done with IRIS GL, but we stuck to our guns and didn't move it way up as some people were  lobbying for us to do. Also, it ended up being higher than what made sense 20 years later.

Indeed, it is higher than what makes sense 20 years later. Remember, it's a contract between implementers and users. The implementations for interactive 3D graphics were a pretty mixed bag back then, and parallelism hadn't really even been exercised in very interesting ways. Memory systems were very bespoke at that point. You had a texture memory and a frame buffer. I mean, they were different things.

The interface needed to be higher in order to have enough room to A, abstract away what had to be beneath and B, leave enough room underneath for implementers to be able to figure out, evolve, experiment, and try things and find what eventually became the more or less standard way of doing these things, so these massively parallel cores on the sharing the cores amongst the different pipeline stages of the engine.

That was not obvious 30 years ago. At least it wasn't obvious to me, but the OpenGL interface was on a high enough abstraction to do that and yet low enough that it could be imperative rather than declarative. An imperative interface is absolutely critical for something you’re going to call an architecture. People have to be able to build things and make things, use it in a functional way, and not just give it directives.

So it was low enough to be imperative but high enough to allow all that experimentation.

Frankly, now 30 years later, the implementation space has matured dramatically, and it's pretty clear what to do down there. Interfaces like Vulcan, harkening back to Khronos, make a great deal of sense because they accomplish, by lowering the interface, they give greater control over the mechanisms that are there; they also broaden the base. They can make compute and graphics all work off a single interface, which OpenGL really would've been pretty tough, I mean, people did GPGPU work, a lot of good work on OpenGL, but it just isn't the right interface long-term for doing compute things.

Because of the maturity of the implementation space, interface moves down, becomes broader, and gives yet more access; it’s much easier to write a game engine on top of a Vulcan that isn't itself trying to manage resources. I mean, the worst thing you can have is to have multiple layers in the same stack competing to manage resources.

OpenGL had to because the definition of those resources needed to be very specialized and managed that way. But again, Vulcan needn't do that in the same way . That's kind of a long-ish discussion of thinking about levels of abstraction. I could go on and on, but I want to come back because I think you asked about what are the appropriate levels of abstraction moving forward.

To answer that, I would say that things like Vulcan are, I think, a wonderful level of abstraction for the architectural level to specify the hardware.

I have almost no experience with the levels of abstraction above that. I've never written code for a game engine. I've probably used that without knowing that I was using them and something. I love the fact that, I mean, OpenGLs kind of this awkward middle ground, by moving down to Vulcan, then you can have the next level, do the resource management, and be a somewhat more descriptive kind of interface that's more about defining data structures and then having things done to those data structures that's almost certainly a more appropriate level for most people to do graphics, right?

I always used to say OpenGL is thoughtfully designed, but it's a power tool. You can cut yourself really easily. I think Vulcan is all the more so, right? Very few people should write Vulcan programs, but because it's a great foundational abstraction, very few people will need to write Vulcan programs on these game engines. Almost everyone who uses interactive graphics will be either coding to a game engine or running on something that was coded to a game engine, and that's all wonderful, but what are the right interfaces for more virtual reality kind of things? I honestly have no opinion about that.

Marc Petit:

Do you think Vulcan had the success it deserves? We have Metal, we have Direct3D, I think we have a little bit more fragmentation... More fragmentation than ever, actually.

Kurt Akeley:

Platform companies like to own APIs. The interface is on their platforms. My understanding is that Apple has succeeded in making it almost impossible to make Vulcan run. Maybe Microsoft hasn't gone to that extreme. It's funny how the tides come in and out on these kinds of behaviors.

I don't know what it should have. I could just say that there are powerful business exigencies that cause Apple to do what Apple does and Microsoft to do what Microsoft does. I'm just so pleased that Neil and the team keep up the good fight and do their best to make it possible to write things that run across.

Marc Petit:

Over the past few years, I mean, ray tracing has been an amazing development in the world of GPU with two layers of acceleration. One, the silicon and the RTX interfaces, and the other one, the AI layer. Where do we go? Can we ever have a real-time path tracer, a proper real-time path tracer? Do you think we can get there with that?

Kurt Akeley: 

From an implementation standpoint, I'll tell a short story and then try to come back to it though.

One of my first professional talks was at the graphics hardware conference in Europe. I don't know, it's like '93, or I forget, '95 maybe, a long time ago. Part of my pitch was to say that this voxelization stuff, that building hardware to do voxel rendering is crazy, that we should just use standard graphics hardware and texture maps to hold the lock holes and render planes. This is a straightforward way to do volume rendering. And, of course, turned out I didn't have any idea, but Ari Kaufman and his students were in the audience there, the audience of 30 people or whatever I thought, so I was definitely preaching heresy to the choir. The funny thing about that, though, is I'm just reviewing; this is how I think.

I like where Nvidia has gone with ray tracing, implementing it as part and parcel of circuits that are also very, very capable of doing sort of forward projection graphics.

The business reason for that is you just never want to leave anyone behind. You want to keep everybody in the tent and add more stuff, but from a technical standpoint, you can think of path tracing or ray tracing as it's sort of a texture mapping function call from a shader. I don't know how exactly they reveal it, but it's certainly possible to do that. If you embed it at that level, then one doesn't need to make this sort of all-or-nothing kind of thing that path tracing versus forward rendering tends to do, rasterization. Instead, we'll say yes, the answer is yes, we'll have this, we'll have that.

All those early ray tracing papers, usually there was only one balance. Like the interactives, there are other interactive, but all they're doing is sort of an awkward kind of rasterization. Over time you get more and more performance, more and more capability, more and more research, and successfully figure out how to get these acceleration structures to actually be interactively updateable.

I'd seen that that was a big problem not too many years ago; maybe that's been addressed as well. You can put all this together to really attractive things where you ray trace where you need to ray trace, and you forward render where you forward render.

If you're designing headlights for cars, you want perfect ray tracing. But if you're an artist, you want a system that has the knobs that you know how to use and that do the things you want to do. Those knobs have evolved over forward rendering, and they're well-implemented there, and at best, there'll be a long transition. I just think architect the interfaces so that they support that transition instead of forcing it to be separated. I think that's what's happening.

Marc Petit:

Another topic is cloud and the Edge with the promise of 5G low-latency networks. What can the cloud and the Edge, what kind of role can they play in interactive graphics?

Kurt Akeley:

I don't know, maybe someone did it before SGI, but we were doing reality servers back in the '90s, and it wasn't called a cloud then; it wasn't in the sense of a moderate Google data warehouse or something, but it was the same kind of idea. I thought I'd use this as an example to bring up another architectural important little observation, which I learned at Stanford, really. That is that   it's nearly free and getting freer, if you will.

The architectural challenge always is about data, having the data you need, where you need it, and when you need it, which often involves moving it around or making multiple copies or whatever. This question of do you compute in the cloud or do you compute at the Edge or do you compute all in the handheld device, which I don't know if that's what you mean by Edge, but it's not like at Google, sort of the Edge of the cloud system.

But whenever, where do you do it? That has less to do with where the compute is expensive and more to do with the data, where the data is expensive to have, and how electric data has to be moved.

One of the jokes I would tell people is that back when I started doing computer graphics, there'd be this handful of polygons, a teeny amount of data, and a 3D graphics pipeline exploded it into a thousand by a thousand pixels. Nowadays, there are terabytes, perhaps more, of data, and a graphics pipeline collectively collapses it down into just a few thousand by a few thousand pixels. It's like the number of pixels stayed almost the same, but what happened is the amount of data behind it is just growing exponentially, insanely. From an architecture standpoint, the decisions one should make is where are the data, where should the data begin, and what's expensive.

Cloud rendering makes sense if you're rendering huge data sets and all you're delivering is just simple things like images. That makes sense to me. And it's consistent with the way, for example, if you're Boeing and you're designing an aircraft, I don't know the numbers, but I can tell you they're big. The number of bytes to describe a modern jet aircraft is just insanely large. They're valuable, by the way. They don't want... I was going to say they don't want McDonnell Douglas, but they bought McDonnell Douglas, so they don't want Airbus to have it.

So it's nicer to keep them in a safe place and just be these really, really small reductions of them into graphical images rather than trying to move the data all out of these unsafe Edge places. That was the SGI thinking there. It all makes sense to me that there'll be more compute where the data are. Maybe for a game, it's easy to have the data be local; probably not, though. I mean, I know nothing about modern game programming, but if it were, then you should do the rendering locally. But if the data is huge, or maybe not.

Marc Petit:

You have the multiplayer problem. The network layer seems to be a bigger problem for scale than the graphics layer at this point. The synchronization.

Kurt Akeley:

Yeah, t he multiplayer, where's the data? All these are more important in a lot of sense than where is the compute cheap because compute just is cheap.

Marc Petit:

This is fascinating, Kurt. We could talk about those things forever. One thing we'd like to do is advice for people who wanted to enter graphics right now.

Kurt Akeley:

I mentioned that when I started working with Jim, I went to the math science library and I read Newman and Sproull, which was kind of the Foley and Van Dam alternate back then. That's how I learned a bit. And then I did stuff.

I would just say doing things, it's unbelievable. I mean, I sit here and program still and what you could learn by just going out to the internet and asking and how quickly you can make something work and how much you learn by just banging through it and making something work. If you want to be able to do graphics, do graphics, right? There's that simplicity. Reading, I would suggest two things. One, which both of them have a teeny self-serving aspect, but I really have to share them, and I think with the right spirit.

One of them is, well, I want to understand how something works in graphics. How does environment mapping work? Because it wasn't part of OpenGL when I did it . Well, it turns out that there is an OpenGL extension for environment mapping, and if you go and read that extension, it's a remarkably powerful document. One reason it's powerful is that it basically says here's how the extension works. Yeah, yeah, okay, but the great thing about those extensions, and by the way, I didn't do this, people did it later, is it put in a kind of history. What were the issues that we ran into? They say we did this while we were developing the early extensions, but then I threw that stuff away. Big mistake. It is so powerful over time to have that history. What are the things we considered? What did we decide and why?

By reading through that history, you learn so much about environment mapping or 3D texture, whatever it is that you're studying.

It's a very, very rich source that's available on the open source history. I think it's almost certainly underused in that regard. It's great stuff.

Then finally, in terms of reading books, I'll put this one up. My name's on here. I actually wrote a chapter in this. That's not the point. The point is John “Spike” Hughes; his name is on there because he wrote somewhat over half of the book, and he did a remarkable job. I think he poured his heart into it and his considerable experience. He is a mathematician by training, and he brings a kind of rigor to it that, frankly, a lot of the earlier computer graphics literature didn't have.

If you want to understand homogeneous coordinates or something, read Spike, read what he says about it; the development is beautiful. I mean, as I said, he put his heart into it. It's too big a book for a course. I mean, you can probably do three courses out of it or something. I suspect it's not used that much. This is why I bring it up. Spike did an amazing job. Just like I read Newman and Sproull, I think there's just an amazing amount to be learned by reading John Hughes.

Marc Petit:

The title of the book for those who do not have the video?

Kurt Akeley:

Computer Graphics: Principles and Practice, 3rd Edition.

Marc Petit:

Even if people are bound to use game engines, I mean, you're so much faster if you understand the underlying mechanism. You don't actually need to learn that level of stuff. But I think understanding it gives you so much insight, and you'll be much better at using a game engine if you understand those concepts.

Patrick Cozzi:

So the book Computer Graphics: Principles and Practice, Morgan McGuire is also one of the authors, and Morgan was on one of the first episodes of the podcast.

Kurt Akeley:

A lot of great people, but Spike just... I mean, he wrote so much of it.

Patrick Cozzi:

I remember waiting in line at SIGGRAPH; many of you signed the book for me, which was very cool. I'm still working my way through it, though. It is a thick book.

Kurt Akeley:

It’s a tome.

Patrick Cozzi:

Well, Kurt, we wanted to wrap up. I kind of had two things. One, I wanted to share a quick story with you.

We talked a lot about OpenGL, and you also just mentioned recently, hey, going online just in general and getting questions answered quickly. I spent a lot of time on opengl.org, the forums back in the 2000s, and I met someone there named Christoph Rico; then we met in person at SIGGRAPH and hit it off really well. We decided that we wanted to do a Gem-style OpenGL book to get a bunch of different authors together.

We went and did that, and we had maybe 40 or so different contributors who either wrote chapters or reviewed the book OpenGL Insights, which I have on the video here. Not trying to plug it. We make almost no money from royalties. I want to say what I learned was the amazing intelligence and passion of everyone that was building with OpenGL.

I would say not only is it an architecture, as you say, not only did it enable an ecosystem in the industry, but the community that it built, the people and how just passion and how much they just loved it and how the OpenGL Banff at SIGGRAPH was like the event of the year for all of us. It's very, very cool.

Marc Petit:

It was really a true catalyst for a community, which that community is what we build the metaverse to.

Kurt Akeley:

Yeah.

Marc Petit:

We build the metaverse upon the work of that community. It's such a foundational role that it has had.

Kurt Akeley:

Wonderful time and we worked hard to make that community. It's wonderful that it happened and that it continues to happen.

Patrick Cozzi:

Kurt, our tradition for wrapping up the episodes is if you'd like to give a shout-out to any person or organization.

Kurt Akeley:

I just want to give that shout-out to Jim Clark. In the end, Jim had the bravery and the drive to create a company; he created an opportunity for so many of us, but in particular for me, to use that to satisfy my desire to build things and understand things and develop things, ultimately, to build the community that you just are describing.

To me, it all traces back to Jim and his inclusive way. Whenever we had an event in those early days, he'd jump up and introduce all the founders, and he started that sense of family amongst the set of us that created Silicon Graphics. I think Silicon Graphics had that sense of family. I think OpenGL has that sense of extended community and family, and it's what makes it all worthwhile to do this. I really appreciate that from Jim.

Marc Petit:

Thank you, Kurt. It's been fantastic to have you. A little bit of a trip down memory lane.

All of this insight also explains why we are where we are today in terms not only of technology but also of mindset. I think it's important. I think we're going through an unprecedented level of collaboration in the industry, and I think that we owe that to going to SIGGRAPH and seeing and doing all of that camaraderie and the importance of working together and being open.

I think we're fortunate that people like you started all of that, like you and Jim Clark. Thank you very much for being there with us. It's been a great moment.

Kurt Akeley:

Thanks again for having me, and thanks for being those people who keep the flame lit.

Marc Petit:

We try. Thank you to our audience. We're delighted to hear from you and to have your feedback.

Until the next episode, thank you very much.