Metaverse Research with Michael Abrash

Announcer:

Today on Building the Open Metaverse.

Michael Abrash:

Our experience of the world comes in through our senses, and our actions are mostly from our hands. And so, if we can drive our senses more accurately and give those experiences, and if we can let our hands be full 25 degree of freedom manipulators, that is really what we are.

Announcer:

Welcome to Building the Open Metaverse, where technology experts discuss how the community is building the open metaverse together. Hosted by Patrick Cozzi from Cesium and Marc Petit from Epic Games.

Marc Petit:

Hello, everyone. And welcome to our show, Building the Open Metaverse. The podcast where technologists share their insights on how the community is building the metaverse together. I'm Marc from Epic Games, and my cohost is Patrick Cozzi from Cesium. Patrick, how are you today?

Patrick Cozzi:

Hey, Marc. I'm super excited for today.

Marc Petit:

Yeah. We're talking today to someone who has seen the early days of basically everything; graphics, programming, CG, VR, and the metaverse. We are super excited to welcome Michael Abrash, Chief Scientist, Reality Labs at Meta. Michael, welcome to the show.

Michael Abrash:

Well, thank you for having me here.

Patrick Cozzi:

And Michael, it's such an honor to have you today. I even brought with me my copy of your book, Zen of Graphics Programming from 1995. I remember reading and struggling through it when I was in high school. I remember your stories on naming Modex to optimizing circle algorithms. And you've worked both as a programmer and a technical writer, on everything from columns for Dr. Dobbs, to Quake, to Valve, and now, your work at Meta. We'd love for you to tell us your journey in your words.

Michael Abrash:

Well, really, I would say that my journey has been a journey of doing what's interesting, and just following that to wherever it led. One of the things that we are all so fortunate to be... We're so fortunate to be in this place where there are so many opportunities, and they're always evolving and there are always new things. I've been doing this for over 40 years, and it's never gotten boring and it's never run out of things to do. And one of the great things about AR, VR in the metaverse is that's going to be true for the next 40 years too. And as a point of contrast, just to see what I mean by that. My first computer was a CPM computer, 56K of memory, four gigahertz Z80. And I only happened to be using it as a total fluke, because, the professor who I was working with at the University of Pennsylvania, had a grant from the Department of Energy to do simulation, but he didn't have free computer time.

Michael Abrash:

And so, I went out and I bought this computer, and I wrote RS32 program and learned about modems, and downloaded my initial effort at the program onto this CPM computer, made the whole thing work, it was successful. And then I had this really cool computer. Now, what's interesting is, that computer to my phone, my phone is a billion times more powerful, literally a billion times the compute. And actually, what's even weirder is my phone has millions of times more storage, which you really would not have thought was going to happen back then. But it really has been just a path of, as things have changed, there are always new challenges and there are always new interesting things. And just keep your eyes open for them, find what you're excited about, and do them. And that's really all it has been. Wish I could say there was a master plan. There wasn't. Somehow, I ended up here.

Marc Petit:

Well, I read that people were seeking you very intensively. John Carmack did, so, there has to be something there.

Michael Abrash:

I can tell you a pretty good story about how I ended up working at id. So, if you'll recall, there was a leaked alpha of Doom, where it was just like one room, and the sprites didn't move, but a friend at work showed it to me, and I was just blown away. And I sent mail to John, and I said, "Wow, that was super cool." I knew John a little bit from an old BBS, before anybody knew who he was. We were talking about 3D. And he sent mail back and said, "Well, my mother lives in Seattle. When I'm up there, we should visit." So, he came up, we had lunch, we talked, he asked me if I wanted to come work at id. I said, "No." I was working on Windows NT, I was leading the development of the graphics. The whole world was basically watching this thing. It was pretty intense.

Michael Abrash:

So, a year later, he came back, and I knew he was going to offer me a job again. And I knew I was going to say no for the same reasons. And we went out to lunch at this Thai restaurant in Bellevue. And for over an hour, he just talked about his vision for, basically, how he was going to build cyberspace. He was talking about persistent servers and client-server, and how people were going to be able to build their own parts of it, and that would basically be the beginning of cyberspace. And by the time he was done, I actually said yes. So, John was very persuasive. And I don't even think he was trying to be. I think he was just saying, "This is how I see the future." And that's sort of how I feel about what we're doing now.

Michael Abrash:

Just so obvious to me that this is what comes next. We've spent 50 years basically developing on the Xerox PARC revolution, where they built the first personal computer, and everything since then has been a much better version of that. And there's been so much that's happened as a result of it, but we still interact with this virtual world through these 2D surfaces. And the obvious next step is, how do we interact with the virtual world, basically, interchangeably with the real world? How do we bring just more useful information to us and make us able to act more effectively in the world? And you think of it, we're CPUs with input and output and a memory. So, that input and output determines what we can do in terms of our experiences, in terms of our effect in the world. And AR and VR are that next step, where all of a sudden, we can use all of our senses, we can use all of our interaction capabilities. So, that's what I'm excited about these days.

Marc Petit:

So, in 2014, when you joined Oculus, Meta, to work with John again. You wrote something that I found. And I remember that, at the time, and I still think it's an interesting quote. You say, "We're on the cusp of what I think is not the next big platform, but rather simply the final platform, the platform to end all platforms. And the past here has been so employable that I can only shake my head." So, how do you feel about that sentence eight years later, as you've been doing this for eight years?

Michael Abrash:

What I meant by the last platform was, basically, what I just described, which is that we have this special case platform now, that brings the virtual world into our world, in this very specific way. And that, once we're moving towards just driving our senses directly, in a broad way, that it's hard to see what comes after that. Now, sure, we could talk about other things in the very long term. I mean, I've seen the matrix too, but realistically, in any reasonable timeframe, it seems to me that our experience of the world comes in through our senses, and our actions are mostly from our hands. And so, if we can drive our senses more accurately and give those experiences, and if we can let our hands be full 25 degree of freedom manipulators, that is really what we are.

Michael Abrash:

And I will point out that I do think there's nothing in physics that says we can't put exactly the right photons in exactly the right place on your retina, and there's nothing in physics that say we can't put the right pressure waves on your ears. Now, there are things in physics that says, "We have no idea how to completely replicate your sense of touch." And there are a whole lot of things that say, "We don't know how we're going to give you full vestibular sensations or taste or smell." There are challenges with all of those. And those are so far off in the future, that it's not something that I really think about, but audio-visual, which is really how we primarily experience the world, I think we can really nail. I just don't know exactly how long it'll take. I see nothing that stops that.

Michael Abrash:

And with haptics, I think what we can do is not make it so that it's exactly the same as the real world, but that we can come up with an alternate type of physics where haptic gloves can give you the ability to be as dexterous as you are. So, if you put your hand down on a virtual table, we can't stop your hand, but we can make it so that you feel that you went through that table and you know it's there. So, I think that there is enough to be done there for a very long time, that we will be mining that space and learning about it. And in a sense, what else is there? That's human experience. So, that's what I meant by this is the last platform. And yeah, it's improbable all the way back to the beginning, whereas I said it was an accident that I even really got into doing this kind of programming.

Marc Petit:

I mean, I think it's obvious to all of us now, after a couple of years of COVID, when we got to experience life through those platforms, but in 2014, it was a pretty big statement. So, thank you for bringing it back. Patrick.

Patrick Cozzi:

So, Michael, you talked about how much compute and storage phones have today, you touched a little bit on AR, VR, and haptics. I mean, if you think about the necessary foundations to achieve the metaverse, is there anything else you want to cover?

Michael Abrash:

So, the metaverse is interesting, in the sense that we're already there. I mean, you can have compelling VR experiences, but it's not like what we've all been through in our lifetime. So, you and I have basically grown up in a world of Moore's Law. Ever since the Alto at Xerox PARC, that platform has not changed fundamentally. You have a way of pointing, you have a way of typing, and you have a 2D screen to present information to you. And what that has meant is that, in conjunction with Moore's Law, everything becomes a software problem, because, you know his new Intel, and then later in video, we're just going to give you more compute power. So, either wait a few years if you can't do it now, or do it now. And that got us out of the habit of thinking about hardware, software stacks that are game-changers, that really fundamentally change the experience.

Michael Abrash:

And the thing about AR and VR is, photons really don't care about Moore's Law, and in a sense, they don't care about your software either. Until not even 10 years ago, it wouldn't matter how good your software was. The best VR headset, you could buy in the world would not give you an experience that's equivalent to what a Quest 2 can give you, not even close. In fact, I've experienced some of those, and it's weird because they'll have like latency of a hundred milliseconds. And you put them on, and the world's just swimming around, for example, or you move your head and they blur, and field of view, wouldn't be... It really required the underlying foundation to move forward.

Michael Abrash:

And I view, really more than anything, our job at Reality Labs research as building the pillars that all that can get built on. So, for example, until somebody builds a display that gives you truly retinal resolution. And I will point out, retinal resolution tends to be called 20/20, 60 pixels per degree. But most humans see better than 20/20. Some humans see 28. And there's also this interesting thing where, unlike the screen you're looking at right now, as you move your head in VR, the pixels actually shift across the underlying pixels. And so, that means you might even need twice as much by Nyquist theorem.

Michael Abrash:

So, till we can get to that very high resolution with a very wide field of view. Definitely, where we are now is enough to give you that immersive experience, but just so you know, that's about a hundred degrees. Your actual field of view right now is at least 220 degrees. So, in terms of giving you the sense of the reality of things around you, proper depth of focus. Can you focus on things at different depths, with as much acuity as you do in the real world? So, all those things need to happen so that we can start having those experiences that really are indistinguishable from reality. This is true across the board in audio. So, the trick in audio is that, the way that you spatialize sound, the way you know where it's coming from, is how it bounces off the pin of your ear and off your shoulder and off your head, and that creates something that's called a head related transfer function. So given someone's head related transfer function, we can do that perfectly. It's amazing. It's like the sound is just coming from those places. Problem is to get one right now, you go and you sit in an anechoic chamber in a very expensive machine for an hour with microphones in your ears while a big arm goes around you making all these noises, which really is not very scalable, I think we could all agree.

Michael Abrash:

So you take your phone and you go like this, and then you have your head related transfer function. That's something that really has to happen in order to give that perfect spatialization. Being with other people, that sense of actually being there with them. An avatar where every time you would meet with someone, you just say, "No, I want to be with you," and an avatar that convinces you that you are there.

Michael Abrash:

And actually, I have an analogy, which is have you ever done something in VR where you look over a cliff? You look at a big drop? That was the moment when I knew VR was a thing, because I did that, and my knees just locked up. I don't like heights, and my knees locked up, and I really wanted to back away. And I eventually made myself step off the edge. And in that split second, before my foot came down, just through my head, there was this utter certainty that I was about to die. I'm wearing a headset and I'm in a room with a carpet. I know where I am consciously. Didn't matter. My body knew what reality was.

Michael Abrash:

And the place that we will ultimately get with other people is that same one, where you actually have to remind yourself, you're not with the person physically. Where you're just accepting it. And that will happen, but that's hard... So all these things have to happen.

Michael Abrash:

And once they all happen, then we are in a very different place. Then we're in the place where real and virtual become interchangeable, where it really becomes possible to do your work remotely and not only be as effective as being there, more effective. So here's one of the things that I'm waiting for, virtual whiteboard compared to a real whiteboard. Real whiteboard, you run out of space, you start erasing. You want to archive, you take a picture of it and hope you can find it later. If you're not facing in the right direction, you have to turn around.

Michael Abrash:

Virtual whiteboard, you want a bigger one, make it bigger. You want more of them, make them bigger or yeah, add more. They're archivable, they're searchable. Everyone can have their own view on... It's a better tool. It's like going from a typewriter to a word processor.

Michael Abrash:

So all this will happen. It's always just a question of how long and what needs to be developed to get to that next level. And that's what we've been trying to do, is look ahead five to 10 years, say what technologies will we need to get to this, and then work backwards and say, how can we get there? It's risky. There's no guarantee about the timeframe, but it has been an exciting ride, and this will happen.

Patrick Cozzi:

That's very inspiring, Michael. My next question, I was going to ask you what you think the platform will ultimately be and how it will change our lives? I think you've already answered that, but if there's any other examples, for the end users, would love to hear them.

Michael Abrash:

Well, one of the things I want to say is that I do not think of myself as being particularly forward looking, much less prophetic. And so the one that I can predict is I can predict the work one. And there are two reasons for that. One is that that's basically what Xerox PARC did. They built something that they used for work because it was useful, and something that I would use. So yeah, that's easy. But I don't really know what the other uses are. And I almost, in a sense... Maybe someone like Alan Kay can see those uses, but what I want to do is I want to build that underlying foundation that millions of people can be creative on. And who knows what they'll come up with, any more than who knew what they would come up with 50 years ago, but I want to enable that to happen. So that's really the goal for me, is just to enable that creativity. And then we'll all find out.

Marc Petit:

So you mentioned human interaction. And I think it's one of the big promise of the technology is high quality and high fidelity human to human interaction. And Meta has been showing some very interesting work on Codec Avatar since 2017. So what's the state of the art in 2022 to reproduce that level of fidelity?

Michael Abrash:

Well, at Connect, I did show the current state of the art. And what I would say is that for faces, I think we're getting across the uncanny valley. Doesn't mean that there isn't a lot of work yet to do, to make sure that it's consistent, that it works for everyone and so forth. But I see that light at the end of the tunnel that says this could really convince you before too long that this is a real person. Bodies are more challenging than faces. There's just so many degrees of freedom there.

Michael Abrash:

And the real challenge is the one of how do you make a person's Codec Avatar. So right now to make a person's Codec Avatar, you have them spend several hours in an environment that's surrounded with cameras, doing different things like going, ah, ah, ah, ah. And obviously, that has to change. That again has to become one of those things where you take a relatively short video, and that's sufficient to create the avatar. So I would say that the state of the art is that we can do pretty compelling things and that we're making progress on bodies as I showed, but that we also, it may be a while before we solve the problem of how do we make your avatar. And that of course is critical, because it has to be your avatar. It's representing you.

Marc Petit:

So, you're thinking about the capture and the photo realism. In the metaverse, things don't have to look like in the real world. So how important are graphical representation versus photo realism? What's your take on this?

Michael Abrash:

So it's really important and it's not that important. So here's the thing. I think that ultimately, having accurate reproduction of the person is really critical for a lot of things. If you're in a business meeting, I don't think you're going to go in as a giant talking hot dog. And I think the people you're meeting with will want the sense that they are actually interacting with the real person. So I think it's very important for a lot of things.

Michael Abrash:

At the same time, people love to accessorize. People love to play with how they look. There is a reason that they do tattoos and piercing and dye their hair and cut their hair and wear different clothes. Except for me, I wear the same clothes every day, but most people wear different clothes, and that will be really important too. And so, the answer is both of those things need to happen.

Michael Abrash:

What's a little bit interesting to me is that... Well, photorealism is the wrong word because they'll all be photorealistic in the sense that they look like they're real, but accurate reproduction of a person, we know what it is, and we know that it has value. Because if we could be doing this in person instead of virtually right now, we'd be doing it. Right? So that one, you can say, here's where we want to get. There's an existence proof and metrics.

Michael Abrash:

Then for stylized ones, you don't know. What level of stylization, what kind, that's going to be an experiment that's going on for the rest of our lifetimes. And actually, it'll probably be one of those things that goes in cycles and waves of how people want to look and so forth. So that in the long run will probably be the more interesting one.

Marc Petit:

Yeah. New fashion on the horizon, the fashion of the look.

Michael Abrash:

Exactly.

Marc Petit:

Okay. So Michael, you just mentioned co-presence is the killer app, the metaverse, the real sense that we are with the person. So how do we know that... And we've been making progress. You mentioned Codec Avatars and how we make progress in facial animation, everything. So how do we know that we have achieved... In sense of an MVP, what do you think is going to be necessary so that we can have that sense of co-presence? And how far away from those capabilities?

Michael Abrash:

I don't know is my answer. And it's one of those things that the only way to find out is to the day that you experience it, you'll say, "Yep, we're there." So, one of the really tough things about AR and VR is that the metrics are all subjective. It's all in how you perceive it. So you don't say... With graphics, it was always what's the resolution? How accurate... You can measure the deviation from ground truth. There's no ground truth to does this give you a sense of co-presence? It's simply one day you're doing it, and you're like, "Oh, wait, I just completely forgot that I was in VR while I was doing it." Which I would imagine has happened to you in VR.

Michael Abrash:

I've had this weird experience where I'll do VR, and later I'll realize there's a memory in my head that I think was real, but it wasn't actually real. It's something I experienced there, but it has landed in my brain the same way. And that's what you want to have happen, where you have the sense of, oh, wait, no, I wasn't with that person. They're on the other side of the country, but the way it feels to you is it's the real thing.

Michael Abrash:

So, I do think that the key to that is really what I'll call biological motion. So I don't think the key is photorealism or accurate reproduction of the person, but it is accurate reproduction of the motion of the person. So, when you're talking with somebody who you know well, there are certain things about how they smile, how they gesture that are just them. And I could easily imagine that you could have a stick figure that you would know as that person, even if they weren't talking.

Michael Abrash:

So capturing that is really much harder than doing the, what I'll call the graphical reproduction, because that involves taking a very limited set of sensors and producing it. So, for VR headset that has downward facing cameras, well, what can those cameras see? They can see your shoulders really well. They can see your hands. But as it gets down, for example, to your legs, it's very hard to know where those feet are. And it's hard to know where hands are when they move out of range, there's occlusions and such. And so, making it so that the person moves like the person, that's what's going to give you that sense of it. And that is a very complex problem that goes from sensors, through machine learning, to graphics. So I don't know if that fully answers your question.

Marc Petit:

That was a... It's indeed a tricky question, and that sense of co-presence is going to be so important. And yeah, I think it is as good as an answer as we could get.

Marc Petit:

The other thing I was curious to hear your opinion on is, like in graphics when we were talking about real time ray tracing and we thought we had to shoot all the rays, but then with AI and denoising we're realizing that a few rays gets you very, very far. And so how much can we rely on AI to help fill those gaps of the sensors? Because as you mentioned, your sensors cannot be always practical or accurate. Can AI completely accelerate us getting there?

Michael Abrash:

So the answer is yes, but first I want to express my... I don't even know what to say. My unhappiness, I guess I'll say, with the fact that's a correct answer. So as someone who started out carefully placing every pixel and manipulating every pixel, and then worked through that to how much more realistic can we get? And really, it's interesting if you go back and look at the original Quake, running at its original resolution, which was 320 by 200, it will look like a jumble of blocks moving around. It's just such low...

Michael Abrash:

Actually, I'll tell you a story about that one, which is that when I was at id, and that was the resolution it was running at, John got an invite from Digital, someone at Digital who had ported quite to a 1K by 1K workstation running at 60 hertz. So we went over and we saw it. And it was unbelievably smooth and unbelievably sharp. And what was interesting was how empty the world suddenly looked because you could see that there wasn't really much in it. Then we went back, I brought up Quake and I'm like, "Oh, something broke." Because it looked so blocky and it looks so choppy. And the thing is, you kind of just get used to the way it's working and then you see the contrast and you can never go back again. But if you look at that and then you contrast it with like a modern first person shooter again, it feels like one of those billion times changes, right?

Michael Abrash:

But all that was basically what I'll call deterministic graphics. You have a description of a scene and you render it as best you can in accordance with the laws of physics. And now we have this element, which is basically like, yeah we just kind of know how to make a better guess. So now we can do it faster with less data and it just feels wrong, but it works really well. And I'll point out that that's really everywhere. So for example, Codec Avatars are all about machine learning. I mean, that's what they are. That's how they can give you this very realistic reproduction from very limited data. They're doing exactly what you're describing. So I think AI really is a core part of the future. I don't think everything will be just machine learned. And the reason is that there are genuine structures underlying things.

Michael Abrash:

Humans have skeletons, for example, and incorporating that is really... It's like an order of magnitude multiplier on what has to be learned. And with machine learning, one of the problems is it gets better when you get more data, but at some point, the amount of data is the number of electrons in the universe, right? And the amount of compute needed is just insane. So I think hybrid systems that are combination of what I would call structured and machine learned are really the future, but there's no question for graphics that it is a fundamental game changer.

Patrick Cozzi:

So Michael, earlier you talked about feeling co-presence in VR. I was wondering, do you think VR will be required for co-presence? Or do you think that AR may also be able to achieve it or even other things like screens?

Michael Abrash:

Well, it's an interesting question with screens. If you could make it feel like you were looking through a window, then perhaps that could work. Your real challenge there is that you need to have a system that can track you properly and can reconstruct you properly. right? And I'm not sure there's a compelling reason to do that on a screen, as opposed to in VR where you really have to do it anyway. So I certainly think it will exist in AR, but I think it's an interesting question, what types of co-presence there will be in AR. And the reason I say that is then VR, fundamentally, I just think of VR as the successor of the personal computer. It's like the rich infrastructure heavy thing where you will always get the best thing that you can do.

Michael Abrash:

Now, 20 years out, AR and VR, maybe they're all merged, maybe there's no difference for technological reasons. I guarantee you 10 years out, that will not be the case. It's just so much harder to do stuff within a very tight weight budget, very tight thermal budget, having to look socially acceptable and having to be able to see through the lenses, right? AR is just facing bigger challenges. No question. I think of VR as the place where we'll do our work. And I mean, something I should have mentioned earlier is I think that's the way the world will do its work. And that actually opens up a lot of opportunity. So I lived in Silicon Valley for four years. I paid an unbelievable amount of money for a house that was basically a beach shack. And it would've been really nice if we had never had to move across the country to go do that, right?

Michael Abrash:

And once you have those workspaces that are more productive than working in person, all of a sudden, everybody in the world has basically the same opportunity and people can really live their lives, where they want and how they want. And I do think that by itself is so transformational. I mean, in some ways it's sort of the reverse of when the car showed up and all of a sudden people could commute, right? And so that meant cities sprawled out. And now what we can do is we make it so, now you don't have to commute and now you don't have to commute from anywhere. So I bring that up partly because I think everybody sees that AR glasses can be the successor of the smartphone, but I don't know that people yet see that VR headsets are really the successor to the personal computer and that it's interesting everybody has a phone, but they're still well over a billion people who use personal computers. And I guarantee you, those personal computers do more productive work than the phones do, right?

Michael Abrash:

And so they're both valuable in different ways. And I want to emphasize that I think VR is the way that work will happen and that is going to just be so transformative. Okay. So the thing about AR is it's always there with you, right? It sees your life the way you see it, to the extent you want to let it do that, and it can help you in many, many different ways because it understands your life. It can do things like automatically adjust the sound level and help pick out the signal from a particular person you're talking to in a noisy environment, it can remind you about things, it can take you places, basically it knows where you are and what you're trying to do. It can estimate your goals and it can help you get there. So it is the thing that is always there with you.

Michael Abrash:

So then the question becomes, what does it mean to be present with other people in that circumstance? Because you could have a whole spectrum. You could have the, what I'll call the heavy end of it, where you're sitting in a room and there's an empty chair across from you. And the person's avatar is sitting in the chair. There's no reason that couldn't happen, but you could also have the one where you're walking down a street and a person's head just kind of pops up floating here next to you and says, hello, and you chat. And then it goes away, right? And is that co-presence? I mean, you're seeing their head, it's like they're right there, but you wouldn't necessarily say the whole body's there because, well, how is the whole body there? There's no consistent way to do it. And also, with just AR glasses, you're not necessarily going to have the cameras to pick up the entire body.

Michael Abrash:

So I think what you'll see is that there's a whole range of what it means to be with other people in that context. And it will develop first in VR, because VR is where you'll have the power and the infrastructure, and realistically at this point, millions of people are using it. So there's really that opportunity. And then pieces of it will move over to AR within those tight constraints. But people are the most interesting thing in the world, right? To other people. And so that will happen. And I think that plus the work stuff... Because really what you can see work is being is collaborative work is what really changes in VR, right? That's why the whiteboard was so important that if you can be with other people brainstorming and you can pop in and pop out, just go ask someone something, it changes everything. Sort of imagine that you're in an office and you want to talk to someone in the office, if you could teleport to where they are, that would be what you would do.

Michael Abrash:

But since you can't, either you spend the time to do the walk or else you just communicate with them in some less effective way. Well, in VR you can teleport, you can always do that. And so it will be such a sea change that we won't even realize that it was a sea change. Just like people so take it for granted today, we're processing spreadsheets, browsers, all that. I mean, it's interesting. I look at my daughter when she's programming and she'll like go To Stack Overflow to look something up, which is great. Stack Overflow really is such an amazing resource. And one of the reasons, it shows what the internet can do at its best, in my opinion. But when I was learning to program, I had one manual or one book and everything else was up to me to figure out, and of course it means progress is slower, but it's a very different world. And it also makes you much more self-reliant and want to understand how things really work.

Michael Abrash:

So one of the fun things then was like scraping down to the metal. It's like, how does this sequence of instructions execute? I mean, I wrote the Zen Timer long, long ago to measure like tiny fragments. And where is the memory, right? How are you accessing it? And now it's built on so many other things that you can't really get down to that. And I do miss that a bit. I mean, it's obviously way more productive. I'm not saying it's a bad thing, but it really was fun feeling like there's the machine, it's yours to optimize.

Marc Petit:

Just want to come back on one question on AR that... You kind of said that the AR glasses would replace the phone, but can they exist without the phone? From a phone factor perspective, do we foresee AR glasses that could stand on their own?

Michael Abrash:

Well, that's one of those things that is going to be an interesting, evolving question. So 5G certainly changes the equation to a significant amount because how much of that compute and thermal load can you offload to edge computing or to the cloud? The answer is, I don't know. And this is where, one of the places where it's useful to talk about the billion time change in the last 40 years or 50 years, because I remember looking at the IBM EGA graphic chip, which was really the first programmable graphic chip. And the place I worked at had popped off the top on one of them. And looking at it, the traces were five microns. You could see them with your naked eye, right? And so looking at that, even though Moore's Law was in fact known at that time and saying, yeah, 40 years from now, we're going to be able to take that, build this instead, this phone, and then it will use like 1000000th the power and it will run all day. It's just like... Again, humans are not good at orders of magnitude and this is an order of magnitude thing.

Michael Abrash:

So when you look at the power and thermals now, they're very tight, right? Then the question is, what will a few decades of research engineering and huge market pressure because people really want it too. Do you remember your first cell phone? My first cell phone looked like a brick, right? And that first personal computer with its 56K, that was... Oh my goodness. So this wasn't mine, this was somebody else's, but it was the first hard disc I'd ever seen. The hard disc was the size of a wash tub. It was five megabytes, not gigabytes, five megabytes. So the thing is you just have to realize that we ride these exponential curves and where it'll be in 10 years, where it'll be in 20 years is probably not even really something we can imagine.

Michael Abrash:

So I think... It's not like I'm saying phones are just going to vanish and certainly not anytime soon, I'm just saying that there's this shift in that direction. And when phones first came out, it certainly didn't mean that people didn't use computers to do all their email and all their messaging and all their... Basically everything, but phone calls, right?

Patrick Cozzi:

So Michael, I wanted to circle back to something from the start of the episode. I mean, we spoke a lot about the visuals and the motion, but to achieve co-presence, there's also a lot with our other sensors and haptics, and I know you have big aspirations there. We're curious, how far do you think it will go?

Michael Abrash:

So haptics specifically?

Marc Petit:

Yeah, we can start with haptics.

Michael Abrash:

So I think that haptics is right on the edge of what is potentially doable, but the effect of being able to use your hands with feedback is so huge that it's worth doing, which is why I have been having us do that for the last eight years. And when we started, I will say that that was really the most blue sky thing that we were doing because there was no plausible path there. I think there's a potentially plausible path there. I certainly can't give you a timeframe and I certainly wouldn't give you any kind of guarantee, but I think that it's possible to put hundreds of sensors on each fingertip. And with that, I think that we can really give you the sense of interacting with real materials, especially if you imagine you put your hand on your desk and you rub it across it, but what you're feeling feels different. And I had an experience that was really interesting with that, because it's not just about the haptics. It's that we're multimodal creatures and we fuse that and fusing it is really what gives us our sense of reality. So I did a demo where there was exactly one actuator on my finger, not hundreds, not tens, just one. And I put on a VR headset and I saw a plate, a brightly colored plate. The kind of, for example, I've seen come from say, Mexico, that it's ceramic, but it's not smooth ceramic. It's rough ceramic that feels almost gritty on the surface. And I rub my finger across the plate... Well, across the table, but it looked like it was across the plate and the actuator vibrated. And I heard the sound of it rubbing across.

Michael Abrash:

And I was rubbing my finger on a plate. There was no question. There was nothing about it that said this isn't real. The funny part is then I shut my eyes and they turned off the audio and I did the same thing. And there was something buzzing on my fingertip. So a lot of your sense of proprioception of what your hands are doing and where they are is actually fused. The proprioception senses are remarkably inaccurate. For example, if you take your hands, you put them outside, shut your eye and try to touch your fingertips, sometimes you can do it, but you don't really know where your hands are. And there's an illusion called the rubber glove illusion, where they basically rub both a dummy of your hand and your actual hand with a feather, but you can only see the dummy because they have a screen up. And then they hit the dummy with a hammer. And it is very convincing at you have transferred ownership to this hand. So you don't really know where your hand is.

Michael Abrash:

So there is every reason to think that because we can provide the audio visual part to go with it, if we can get those hundreds of actuators on your fingertips and give some resistance. There are techniques can potentially make it so that when you close your hand, you can feel resistance. The thing is, we just can't keep your hand from moving. But I think that we really can give you those experiences where for example, your keyboards virtual and that's fine. And actually it's a cool thing. If you had a virtual keyboard with that, your hands aren't locked to where the keys are. If your hands drift around some, we should be able to figure out where you think the keys are now. And so potentially you could type with a lot less error and a lot faster, because it could be your own personal keyboard that was configured exactly where you wanted it.

Michael Abrash:

So I am very excited about the potential for the haptic work we're doing. At the same time, I am patient because it's not showing up anytime soon, if ever.

Marc Petit:

How important is smell?

Michael Abrash:

So I would love to do smell. And that's something we looked at early on. Smell is one of the senses that isn't mediated. It goes directly into your brain, which is part of why it's so strongly associated with memories. And if we could do smell, that would be great. Having looked into the state of things with smell, it is clear to me that there's no plausible path within 10 years to something that could potentially be shipped. And that's where I draw the line. It's like, you have to be able to say, well, if everything goes right, here's the path that we could go down that could have this going into a product within 10 years. Because out past 10 years, I think you're just kind of guessing. Technology can pass you by. And so I'd love to do it. Couldn't figure out a way to do it. After this podcast, probably someone will get in touch with me and say, oh, we have solved that problem as people do periodically, but we still haven't found the case where that happens.

Marc Petit:

I know two companies in Montreal actually working on this. So it's interesting.

Michael Abrash:

I hope they're successful. I would love that.

Marc Petit:

Yeah. It's interesting. Well, Michael, we're going to go to the closing questions, but thanks for pulling your crystal ball out for us. I think that's what interests all of us, trying to calibrate what's likely to be there soon, what's going to take time. So I know it's a difficult guessing game, but thanks for rolling with the punches with us today.

Michael Abrash:

This has been fun. And I mean, I do want to say that really the big thing in my crystal ball, which is not very accurate in any near term detail, but the big thing is just this is coming because it is inevitable. I mean, this is just the evolution of what we've been doing. This is what human oriented computing needs to be to put humans at the center of this whole experience. And so being able to be with other people, having technology that's responsive to your needs, being able to use all your senses, use your control. I mean, this is what we are. So this is the reason I'm doing it. It's hard to imagine, it's a once in a lifetime chance.

Marc Petit:

Yep. And it will happen. So I'll take the first one. Is there any topic that we did not discuss today, but we should have?

Michael Abrash:

Actually, I'd like to talk about one technology that I am excited about, which is electromyography, EMG. So electromyography involves sensors on your wrist that can detect the nerve impulses going through the motor neurons to your fingers. And I think electromyography has the potential to be the mouse in the keyboard of AR and ultimately a VR. I should talk about that. So if you move your fingers like this, it's easy to replicate. If you move just a millimeter, it can pick that up. There'll be a lot of research to get to that point. But I think it's possible that not only will it be useful for doing like taps and clicks and scrolls and such, but that potentially it could be used to do typing. And if you think about that versus having to take out a phone and type on the keyboard, that would be pretty exciting to have something on your wrist that you can be typing at speed with.

Michael Abrash:

So electromyography is really kind of a novel thing. And I think it is like that piece of the puzzle for AR because how do you control it? You have these great glasses, let's say. How do you interact with them? How do you control what you want to do? The other piece though is personalized contextualized AI. Because again, these glasses see whatever of your life you want them to. And so they can start to understand your goals, your needs, the context you're in. And they can start to proactively try to assist you. So for the first time, it really does become that assistant is always there trying to help you do what you want to do. Combine that with EMG, because of course it can't read your mind. So there's this very lightweight way for you to give it feedback or to answer questions. And I think it'll be a lot like when the GUI showed up.

Michael Abrash:

When Doug Engelbart and then Xerox PARC created the GUI, which we're living with now everywhere. My car has it for crying out loud. But we need new interface for AR because it has to be an interface that works everywhere, that works with you. That's very low friction. And ultimately, that will be the interface for VR, as well because VR, you're just going to be in a virtual world. The metaverse is a virtual world.

Michael Abrash:

So you're still going to want to do the same things. So that I wanted to bring up. And the other thing that I did want to mention is that the metaverse is coming. I think it's very clear that Meta is very committed to making this happen. But I also do want to say, because I think it's important to emphasize, we are not trying to do this alone. We're not interested in it. There's too much to do here. And it needs to be something that is open and that is an effort across the whole community. Just like no one company made personal computing happen. I just love having other people excited about this and working on it. And the other thing, which really isn't like my first thought, but I think it's something I should mention is we certainly are hiring people and we have lots of interesting roles. So people who are interested, we would love to hear from you. Okay. That was my public service announcement.

Marc Petit:

All of that was music to our ears.

Patrick Cozzi:

Yes, we love the open theme. We love the collaboration theme. Yeah. Really fantastic. And Michael, to wrap things up, is there any individual or organization you want to give a shout out to?

Michael Abrash:

Well, I mean, as I say, it's hard to imagine that I'm in this position to be able to enable this technology to get developed for the future. And I really have to say that it has been remarkable being at Facebook and then Meta to see the commitment to creating this future. This degree of basically future looking and willingness to invest is something I have never really seen anywhere before. And I feel unbelievably fortunate to be at the leading edge of that.

Marc Petit:

Well, Michael, thank you. We were unbelievably fortunate to have you with us today, sharing all of your insights. So thank you so much for taking the time. We know how busy you guys are at Meta. So I want to thank Patrick as well to be with us today and also want to thank our audience. Thanks to people like you, Michael, we get a lot more listeners. We get a lot of questions. So to the folks who listen to us, thank you so much and hit us on social with questions, suggestions, and we'll be back for the next episode of Building the Open Metaverse. Thank you, Michael. Thank you, Patrick.

Michael Abrash:

Thank you, Marc and Patrick. This has been so much fun and thanks for having me.

Marc Petit:

You're absolutely welcome.