Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Today we're talking graphics, the intersection of AI and graphics specifically, and maybe a little bit on graphics in augmented and virtual reality. and how AI is proving a vital component of making consumer augmented reality happen.
There may be no better guest to talk AI and graphics than our guest today, David is vice president of graphics research at NVIDIA.
He co-founded NVIDIA Research in 2006 after eight years on the faculty at the University of Virginia.
His research interests include AR and VR, ray tracing, real-time rendering, GPU computing, and the intersection of machine learning with computer graphics.
David was also recently elected as a fellow of the IEEE for contributions to GPU computing and computer graphics.
David, thanks so much for taking the time to join the AI podcast.
Yeah, thanks for having me. So we've got SIGGRAPH coming up, or when people are listening to this, it may be in flight online as everything is this year with conferences going virtual.
So I know there's some stuff related to that to talk about, but let's kind of start at the top with what we tease, the intersection of AI and graphics. all the work that you and your colleagues are doing at NVIDIA Research around this topic.
We were talking before we came on the air, and you mentioned there are kind of two basic forks to this work.
So I'm going to kind of leave it to you to get us going talking AI and graphics.
Sure. So It's an interesting dichotomy or sort of a false dichotomy, but it's a very natural one.
So most people, five, 10 years ago, if you'd asked people what Nvidia did, people would have said, oh, computer graphics, computer gaming, right?
That's sort of our DNA. That's our origin.
And so people associate Nvidia with graphics and gaming.
And that's certainly right there in our DNA.
On the other hand, there's a whole sort of category of people who, if you ask them what Nvidia does, they think of us as the company who builds the hardware that's powering this AI revolution, right?
If you talk to data scientists or deep learning Folks, they probably know that Nvidia's processors are used for graphics and gaming, but that's not what they care about.
That's not what they use on a day-to-day basis.
So I think a really interesting question and one that I was asking myself, again, you know, five, four or five years ago is do these goals conflict, right?
Do AI and Computer graphics, are they fundamentally different end cases, different use cases? different end games for GPUs.
In fact, is our architecture going to have to split to cover both these markets?
And what's been really interesting is I've had this realization over time that actually, no, AI and computer graphics go together like peanut butter and jelly.
In fact, artificial intelligence, deep neural networks, that is the future of computer graphics.
And you see this today, you know, not just at Nvidia, but in SIGGRAPH, you mentioned SIGGRAPH.
If you just go to the SIGGRAPH, you know, proceedings, all the technical papers that are the cream of the crop of computer graphics research.
And look at the titles, probably a third of them have deep in the name.
The field, like other fields, like computer vision, like natural language processing, is absolutely getting revolutionized by AI.
In fact, the parts of our chip that work on tensor calculations, the tensor cores that accelerate deep learning, the parts of our chip that accelerate ray tracing, that accelerate rasterization and texture mapping, and all these aspects of computer graphics they go together beautifully.
They mesh beautifully and we can use them to do things we could never do before in computer graphics and in image synthesis.
So you brought up this question of sort of two different forks where the intersection of AI and computer graphics, there's kind of two major landmasses on that map.
And one is there's a whole category of work where you're using AI or a deep neural network in some form to either replace or augment components of the traditional computer graphics pipeline, right?
So that might be way at the beginning of the pipeline where you're using AI for content creation to generate meshes or to do a realistic character animation or to capture materials.
For example, our team had a paper several years ago that was on you know, capturing realistic materials, whether that would be leather or denim or wood or, you know, the glossy cover of a book, you know, from a single iPhone photo.
That's an example of something where it used deep neural networks to accelerate material acquisition.
That's all at the very front of the pipeline.
So I know you had my colleague, Aaron Lafon, on recently.
And Aaron's talked a lot about something happening at the very end of the pipeline.
One of our big successes with AI and computer graphics is DLSS. this sort of fusion of anti-aliasing and super sampling that lets us get better image quality and faster frame rates by rendering a low res image accumulating results from several frames and, you know, using that to paint a high resolution image for this frame.
Right. That sits at the very end of the frame, but it's fundamentally sort of a bolt-on.
It's a component that you add on to the existing graphics pipeline.
So that's one continent, right, in this sort of space of how do you merge AI and graphics.
Quick shout out or plug. Aaron was terrific.
No surprise to you, I'm sure, Dave. But if you haven't listened to Aaron's podcast yet out there and you're interested in the graphics pipeline and how you know, AI is shaping things highly recommend checking that out as well.
Sorry to interrupt. No, absolutely. Aaron's group is, is leading, leading the way, uh, in this research and is working with all kinds of other groups, product groups, advanced research and development groups at NVIDIA and really making sure they take their research all the way sort of over the finish line and taking it that last mile to, to make, not just, you know, cool papers that appear in SIGGRAPH, but incredible features that animate our products and, you know, sort of bring more value ultimately to the customer.
Absolutely. And DLSS is a perfect example of that.
And it's also a perfect example of this concept of augmenting the graphics pipeline, either by replacing something or adding something to it.
The other side of AI and computer graphics is this much more ambitious vision of a neural network that just directly synthesizes the image.
And so I call this direct neural synthesis or neural image synthesis.
And this is... a incredibly prolific field right now, but it's largely kind of grown out of the computer vision community.
It's not actually being actively used very much in graphics yet.
But this is a place where I spend a lot of my personal time and sort of research energy.
And at one extreme is something... A good example is something that came out of NVIDIA last year called StyleGAN.
Right, right, right. And the latest version of that is StyleGAN2.
So if you're, if you're out there Googling it, you know, Google StyleGAN2 and you'll see all these incredible images that we, but also lots of others, lots of artists and researchers have picked up and run with.
StyleGAN2 is the extreme version. You've got this neural network, And you put a bunch of random numbers in and what comes out is a high resolution image.
And if you train this neural network with, you know, good quality, high resolution images and enough of them, then you can get incredible results.
So, you know, realistic pictures of people or cats or dogs or artistic images that are trained not on photographs of people, but on, say, artistic portraits of people.
At the CVPR Computer Vision Conference over the summer, we released a video showcasing some of this sort of artistic use of StyleGAN, where we trained it on you know, thousands of images from artists and created a sort of interactive exploration tool where you can you know, sort of dynamically choose the content of a portrait.
So is this a man or a woman? Do they have long hair, short hair?
Do they have glasses? But also the artistic style is more like a charcoal sketch or a watercolor and oil painting.
And you can sort of dynamically and fluidly sort of explore content and style at the same time.
Very cool. So I want to go back for a second and kind of for folks listening along and trying to imagine, when you said that As input, you give StyleGAN random numbers.
Is it literally just random numbers or do the numbers or even how you generate the random numbers have anything to do with the output.
So StyleGAN is this extreme example of unconditional image synthesis, where it is literally just a string of random numbers between zero and one or negative one and one.
And in our case, there's 512 of them. Okay.
So 512 numbers go in and a megapixel image comes out.
And if it's been trained on high quality data and enough of it, then these images that come out can be breathtakingly realistic, far more realistic than we really know how to do approaching normal real-time graphics.
I'd say our images, the best of the images generated by StyleGAN are as good as something that would be generated by Hollywood with hours of time.
Oh, and that's another interesting aspect I should mention.
So the process of inference, right? Running the rant, you know, taking those random numbers and sort of pushing them through the network and generate megapixel image.
When we did this before on the Volta architecture, it took 24 milliseconds.
And our new Ampere architecture, I haven't even measured it yet, but I'm sure it's considerably faster than that.
So it's very fast. Well, that was one of the things you mentioned, Aaron Lafon on the show and just keep plugging that episode, but it was great.
You know, he was talking about, you know, the Hollywood corollary sort of and how, you And a movie could take, you know, hours, days, weeks, however long.
But in their realm, you know, particularly with online gaming – or not online, but with gaming – you need to do it in real time, right?
You've got a 60th of a second to get that next frame generated.
It's a theme of the show, and you know this better than anybody being in NVIDIA research for so long, that the acceleration in the past five years— of the hardware and then what that can do to using ML and AI and research in real world is just the thing that everybody keeps talking about.
It really is something. Yeah, there's, you know, Aaron focuses on real-time rendering research, sort of these constraints.
As you said, you have to generate an image in 16 milliseconds.
And that's everything. So if you have a particular effect that you're trying to get, or you're trying to make a particular character or something, you maybe have two or three milliseconds.
So it's an incredibly compressed timeline.
And it's just astonishing to me that we've been able to take some of these techniques And through a combination of hardware optimization, software optimization, faster GPUs, faster tensor cores, clever algorithmic insights,
You know, in some of our experiments we've gotten from sort of the initial proof of concept, I'm thinking of DLSS, the first version of that, you know, ran in PyTorch or some framework like that run in Python.
And the final version is over a thousand times faster.
So it does take effort, But yeah, the speed of the computers that we have these days, the speed of the GPUs that we have these days is really breathtaking.
So at one end of the spectrum, you talked about putting random numbers in, 512 random numbers in.
And if you've got good enough data, incredibly lifelike photorealistic image comes out.
What's at the other end of the spectrum?
Right, because the problem, of course, is 512 random numbers doesn't doesn't sound like it gives you a lot of control.
And so what I described as unconditional image synthesis, you know, random numbers in, image out, But it turns out that those random numbers do have some correlation to what's going in.
You can smoothly change the random numbers, just sort of slowly vary them, and the image will change in sort of smooth and often kind of intuitive ways.
So there's been a lot of work to sort of study what they call the latent space of these GANs.
So if you think of the latent space would be this sort of 512 dimensional space defined by those 512 numbers, right?
So if a given point in that space you know, in that, in that high dimensional space will generate a given image.
What happens is you slide the point around, you know, in all of those 512 dimensions.
Yeah. And it turns out that you can kind of study this after the fact and do some principal component analysis and extract things like, oh, if I kind of move that point in this direction in that high dimensional space, then the person, the photograph will start to look older, right?
You know, or the person will start to smile or the person, or, you know, there is sort of a gender directions where you can sort of slide from, you know, a person. that has one facial identity, but you can kind of slide them from sort of male to female or from old to young.
You can do this with Horses, you know, pictures of horses.
You can slide from sort of, you know, there's a horse galloping by itself to a horse with a rider.
And at some point you sort of cross the threshold and a rider kind of pops up and appears on it.
Or cars is a better example. Maybe you can find the sort of set of directions in latent space, the set of changes that you need to make to take an image and sort of make it look more like a sports car or more like a truck.
In addition, of course, to being able to change the apparent camera view.
And this one is really spooky to a computer graphics person like myself, because we've grown up with all these ideas of how you do computer graphics and involves four by four transformation matrices or ray tracing or things like that.
And here we've just shown tens of thousands of cars through a neural network And it has, like a Renaissance artist, it has kind of puzzled out the rules of perspective and rotation.
I can go in there and find the ways I can move that latent point, that point in latent space, to change the apparent camera angle, to look at the same car from a different direction.
So there is a rudimentary amount of control right there in the latent space.
And there's been a lot of work by us and by others on sort of exploring that latent space and finding out how you sort of find and control it after the fact.
On the other end of the spectrum is something where you tell the neural network, for every single pixel, what you want to see at that pixel.
And so this is most famously illustrated by NVIDIA's Gauguin algorithm.
Gauguin, you give it what looks like sort of a paint-by-numbers map.
These pixels have rocks and these pixels have rocks. sky and these pixels have water and these pixels have trees and these pixels have beach.
And it turns that into a photorealistic landscape that respects that incoming what we call a semantic segmentation, right?
That sort of grouping of the image into large blocks of pixels that are just kind of like, this is tree, this is car, this is road, this is building.
So that's sort of the other end of the spectrum.
That's conditional image synthesis, where you have conditioned the output you want on some particular input.
And In between those spaces is where a lot of the action is right now.
You'd like to be able to control things with even less incoming data.
So instead of having to paint for every pixel, whether I want a beach, I'd like to be able to say, just kind of give me a tree on the beach in front of a mountain. you know, the sailboat to the left, no, a little further left, right?
You know, you'd like to have that kind of conversation with your system.
You can imagine, you know, using natural language processing and melding it.
But an artist, a real artist, who does this professionally is going to want far more control at times too.
They're going to want to be able to get in there and adjust each individual pixel as well as higher level controls, such as lighting and mood and tone and so forth.
So this aspect of control over these neural networks is I think one of the places we're really focusing, we and others in the community are really focusing, is how do you control the output of these neural networks to produce exactly the image that the artist has in mind.
Is there one end of the spectrum or even a point somewhere along the way that as a researcher is I don't know, more surprising to you, or if you look back to, you know, 06, when you joined NVIDIA, if you could put yourself there. and then know what was coming to think like, wow, I never would have thought, you know, they use the word spooky before.
I never would have thought something this spooky was coming.
Or is it all just kind of... Like, you know, to use the word spectrum again, it is on a spectrum.
No, no. There's definitely been surprises.
And I'll go back to that. The first time I saw that video... that was made by members of my team, a team of deep learning and graphics researchers in Helsinki, Finland.
They created the StyleGAN, video and they fed in all these pictures of cars and then they just sort of like wandered the latent space or Brownian motion within that latent space.
And you saw these cars, you change angle.
You looked at it from a different angle.
And I had this sudden realization that we were it sent chills down my spine to see like perspective and rotation being more or less correctly approximated. by a neural network that knew nothing of the mathematics, right?
All the things we teach people in Graphics 101, all the things I taught people as a university professor, just being sort of sidestepped here And the results look extremely plausible.
So that was a deep surprise to me. I was absolutely flabbergasted by that.
I'd say... In terms of AI and computer graphics, the thing that consistently surprises me is just how effective deep neural networks are.
I'll give one more example, perhaps. There's some work that actually is a collaboration between Aaron LaFond's team, to go back to the podcast that Aaron did recently, And Alex Keller, who's another one of my directors and his team, their teams have a collaboration in Zurich where they are training what's called the light field. a neural network to represent the light field.
So the light field, if you could think of a three-dimensional scene, for any point in that scene, you should be able to describe what color light is going in all directions.
So if you just pick a point in the open air, then, you know, light is passing through it, you know, from the ceiling down toward the floor, that's sort of ceiling colored and from the floor up to the ceiling and that's floor colored. you know, and so forth.
And so you can imagine this sort of function that takes, you know, a point in space, so X, Y, Z, and a direction, so, you know, theta phi, you know, two to, you know, azimuth and elevation.
So sort of five-dimensional function that will tell you in any of the, you know, in any of those directions at any of those points, what the color is, what the light is doing there.
That's the light field. And it turns out you can train a neural network to represent that light field amazingly effectively.
And then you can use that approximation that neural networks approximation of the light field to train or to, to better sample it, to do the ray tracing and the path tracing. that Aaron alluded to his podcast better and to do it faster.
And that's one of the really Really surprising finds to me is that the neural network has enough capacity and sort of enough ability to approximate and estimate what's going on. that it can actually capture an entire light field of a reasonably complex scene.
Our guest today is David Lipke. David is Vice President of Graphics Research at NVIDIA.
He's been with the company since he co-founded NVIDIA Research in 2006 after, as he mentioned, teaching at the University of Virginia.
We're talking about all kinds of things related to graphics and AI.
We're going to switch gears for a second here and talk a little bit about augmented and virtual reality.
David, I know that there is a keynote from SID Display Week, and I believe you gave it again at SIGGRAPH this year, talking about display technology.
Can you talk a little bit about that and how it's impacting augmented reality in particular?
And there's been a lot of talk about consumer augmented reality.
You can get it on your phone these days, but glasses possibly coming down the And I'm not sure exactly where your research fits into all of that, but just to paint the picture of what augmented reality might be and how...
AI is impacting and pushing us towards that maybe being a reality soon.
Yeah, so I would say that virtual and augmented reality has been a long time interest of mine.
I suspect most people who are in the field of computer graphics have been inspired to a greater or lesser degree, by the vision of virtual reality that Ivan Sutherland, who founded our field, was the first to describe it.
He had this famous seminal talk called The Ultimate Display, where he sort of laid out the whole concept of a display that was So realistic that you would feel like you were transported to this virtual world.
Right. And so, you know, so I, like many other people was just deeply inspired by that.
And it's really that, that vision that, that. brought me to computer graphics.
I was a chemistry major in college and was sort of doing other things.
And it was that vision of virtual reality that took me I put a left turn on my career path and brought me into this field.
But it turns out that there's many things that are hard about virtual and augmented reality. especially the sort of the vision that you get in the media and the movies, the vision that everybody wants of of something as sort of vanishingly unobtrusive as a pair of sunglasses, right?
That's what we all want. I want augmented reality to look like, oh, you know, the movie Kingsman or, or some, or your princess Leia, right?
I want it to be completely unabashed. And, uh, There's many things that are hard about that, but I think one of the biggest stumbling blocks is the actual display technology.
Many years ago, we started working on display technology at NVIDIA Research.
It's a sensible thing for NVIDIA to be concerned with because even though At the time, we didn't make displays.
This is before our G-Sync line of displays. we always knew that displays were kind of ultimately the customers for our pixels, right?
You know, GPUs talk to displays. If displays are going to change if they're going to get much higher resolution or much higher frame rate, or at the end of this, maybe I'll pull this back to AI, if they're going to have to display holograms, actual wave optics, holograms, instead of just normal amplitude images.
We need to know that. NVIDIA needs to know that, and we need to know that before other people know it.
And so we've had this sort of long-standing mission to look look down the road in terms of display technology and in terms of use cases and augmented virtual reality are right at that intersection.
Virtual reality is unique. You need what you want.
To make it compelling and immersive, you want very high frame rate. because you need a very smooth image and you want very low latency, right?
When you move your head, the thing you're looking at has to appear to change, you know, almost instantly, you know, so you need, you know, faster than 60 frames per second, ideally up in the 100, 120 frames per second range.
So you need very high frame rate, You also need a very large pixel count, very high pixel density because the human eye is this exquisitely detailed instrument.
You know, we can resolve one minute of arc, right?
60 cycles per degree. And the way I think about that is that a degree is roughly the width of your thumb at arm's length.
And so imagine painting 60 little black and white stripes you know, alternating black and white stripes on your thumbnail at arm's length is about where those 60 stripes would sort of blend into just a uniform gray where you wouldn't be able to resolve the stripes.
That's the level of detail that your eye has.
You can see something that detailed, but it also has a huge, like 180 degree field of view.
If you work through the math and different people plug in different values, but if you wanted to support that level of resolution, across the entire field of view, you need something like 21,000 pixels wide and, you know, 18,000 pixels high or something, you know, vastly More pixels than any display you can buy today.
Certainly anything that you can put on your face and wear.
And then your eyes can focus at different lengths.
Maybe you need eyeglasses and you need to incorporate your eyeglasses prescription or your contact lens prescription into the display directly.
Building pixels for humans is kind of how I think of the display technology mission.
And it becomes, it's really acute for augmented virtual reality.
So now that we've got an understanding of what's involved in just the complicated world of making these images right on the human eye that are sophisticated enough to sort of trick our brains into thinking we're in a different reality.
How does AI play into this? There's a lot of places where AI can play into this.
First of all, I sort of alluded to the idea that you probably can't afford to render all those pixels across that whole field of view.
Right. There's just too many pixels. Right.
You're going to end up with what they call foveated display.
And the fovea is the part of the eye that's high resolution.
And it turns out it's only roughly the central four degrees of your vision is really high resolution.
The periphery, the peripheral vision, is low resolution, but it's very sensitive to flicker.
It's very sensitive to temporal change. And so you need something which is fast, and very high resolution, but only in some places.
And where those places are change, and of course they change as your eye moves around.
Where I'm going with this is eye tracking.
I think one of the important ingredients to getting, let's say, satisfying virtual and augmented reality you know, maybe not the first set of products that come out, but the eventual products that we all want to use.
I'm absolutely convinced we'll use eye tracking to know, they will know where your gaze is in terms of the image that you're looking at.
And we'll use that for all kinds of things. including foveated rendering, you know, using high resolution where you're looking and rendering with high resolution, displaying with high resolution where you're looking and low resolution in your periphery.
Right. And AI, I think is a very useful.
So, you know, gaze tracking is fundamentally a computer vision task, right?
You have cameras or some sort of sensor looking at the eye And it turns out that AI is very useful for this.
So one paper that my group produced a couple of years ago was essentially a training protocol and a training data set for training a neural network to take a camera image of the eye and figure out, you know, and map that to a location on the screen, map that to where that eye is looking.
Case tracking is one example of where AI comes into it.
Augmented reality, you know, the full sort of the full on vision of it, not just sort of, you know, calendar notifications and texts popping up, you know, like Google glass style in your, like a rear view mirror. in your vision, but actual synthetic objects embedded in the real world, that's going to require understanding the real world.
And that's classic computer vision, only it has to be done really fast And, you know, in a device on your face that, you know, you know, can have maybe half a watt of, of total power consumption, right.
Between the display and all the computation.
So very low power, very fast computer vision, you know, scene understanding, scene reconstruction.
So that's clearly an AI problem. And the rendering itself, and this is less obvious, but it kind of goes back to what I said at the beginning, that AI and computer graphics are turning out to kind of supercharge each other.
So a great example of this is holographic displays.
I briefly alluded to the fact that your eyes have adaptive focus, right?
Your eyes focus at different depths. It turns out that that's actually an important cue.
It's not the biggest cue. It's not the one that you notice first.
But if you go to a 3D movie or you have a 3D television, or you wear a pair of virtual reality headset, a virtual reality headset for a while, Your eyes do get fatigued.
They get eye strain. It's because they're having to focus at one distance while While what's called the vergence, which is the crossing of the ways your eyes move in their sockets in order to sort of triangulate and point at whatever thing close or near they're looking at.
The vergence is conflicting with the accommodation, which is the actual focus.
So this vergence accommodation conflict is one of the, I think, really important parts of getting virtual reality, augmented reality, getting it right, making it comfortable, make it something you're going to want to use all day long.
Is it that the way that depth in something like a 3D movie or a VR headset is created, is it sort of exploiting that conflict?
I would say that you would like to make that conflict go away.
So let's take an example of sitting there watching a 3D movie, but watching it on a television.
Right. The screen is maybe six, 10 feet away from you, but your eyes are focused at 10 feet away or else you get a blurry image, right?
So your eyes have to stay focused at that 10 foot distance.
But it is displaying different images to your left and right eyes such that if you want to focus on something... I shouldn't say focus.
If you want to gaze at something in the foreground, then... your eyes kind of cross a little bit so that they're converging on a point that is in front of the television.
And if they want to show you something in the background, then your eyes diverge a little bit and you're gazing now at something that is behind the television.
So the vergence, which is the movement of your eyeballs and their sockets, is conflicting with the accommodation, which is the focus, the actual focus mechanism of your eye, the crystalline lens deforming to, you know, your eye to the correct focus.
Here's a tip for the listener at home. You can rewind 30 seconds and then do what I'm doing right now, which is actually staring at a monitor. and trying to do with my eyes what David, what you're describing.
And it actually helps to understand this for the uninitiated.
Yeah, if we had video, you would see me holding up my finger in front of my face and kind of moving it closer and further away.
And my eyes cross as my finger gets close to my nose.
Even better. There's this conflict inherent in any 3D TV.
It's also inherent in any VR headset that's been built to date or that's been sold to date.
Right. All of them have, you know, sort of optics in the package that make the image appear to be usually between six to 10 feet away, sort of. half a diopter away if you want to think of it like an optician.
And then your, your eyes have to focus at that say six to 10 foot range.
But in the meantime, they're showing you stuff that is supposed to be both closer and further.
And so your eyes have to cross or uncross adaptively.
So there's Burgeon's accommodation conflict.
One of the most promising ways to deal with this vergence accommodation conflict is actually to do a hologram.
And by this, I mean a real hologram with lasers bouncing off of a special chip that changes the phase of the laser.
The details aren't important, but the point is this is not the Microsoft HoloLens sort of co-opting of the term hologram to mean just like a 3D icon.
This is an actual... honest to God display that is making the appearance, making a 3D image appear to float in midair.
And that sounds like science fiction, but actually you can build it.
It's not even all that complicated. What is complicated is computing exactly what to set all of those phases of the little pixels on that holographic display, how to set all those phases.
You're actually having to do this very complicated wave optics math to do this computation.
And this pulls it back to AI because it turns out that there is this extremely complicated Well, it's not really.
It's Maxwell's equations for the physicists out there.
You could take Maxwell's equations and plug them into a solver and kind of figure out what the phase needs to be at every pixel of a holographic display to create a particular image, a particular light field.
But that's really expensive. So what we do is we do that and we train a neural network to approximate that phase image and we can get incredibly realistic three-dimensional images by illuminating a holographic display with a laser and driving that display with a neural network in real time.
And so some of the most exciting work that's happening right now in this field is happening at Stanford.
We happen to be collaborating on this project with Stanford that they're leading that is actually training a neural network to drive a holographic display by pointing a camera at that holographic display and taking the output of the camera and feeding it directly into the neural network. so that it learns how to create a high-quality image from these lasers bouncing off this, what's called a SLM, a spatial light modulator that actually adjusts the phase at every pixel.
So that's, from a very prosaic use of AI on one end to simply do the eye tracking, pretty conventional computer vision application of deep neural networks. all the way to something like very exotic, but actually maybe not as far off as people think to drive a holographic display that would ultimately be perhaps the best way to actually get an image directly into your eye in a lightweight pair of augmented reality smart glasses.
I was going to say, so if we can... And I say we.
Really? I mean, it's you and your colleagues.
Train these networks to do a good job of that and then shrink the whole thing down into a lightweight, comfortable package that only takes half a watt total power.
Right. Then we've got super high quality consumer AR.
It's coming someday. It's going to be a few steps.
So David, this has been great. We've covered a lot of ground and there's a lot of meat in here for the technical and non-technical as well that you're on, if you will.
So this has been... Wonderful conversation.
We like to land the podcast on a little bit of a forward-looking note.
Not that everything we've talked about hasn't been forward-looking.
But going back to kind of what you set up is these sort of two, these two forks to where this is all headed.
And you've got, you know, using AI as sort of, I think he used the term bolt-on at the end of the graphics pipeline versus, you know, synthesizing or just generating, skipping the whole pipeline, as you said, and using AI to generate your imagery kind of from scratch.
Going forward, what what do you think or what's your vision for how these two different approaches are going to kind of be combined or or just how AI is going to shape the way that we look at and create graphics going forward.
Right. So we talked about these sort of different approaches of, you know, either using a neural network to augment or replace parts of the graphics process, the graphics pipeline. versus just a direct synthesis where you take some random numbers or maybe a paint by numbers image. and synthesizing something, a direct image synthesis.
So I do think that we're going to meet in the middle.
Right, that actually the interesting space is probably between these two extremes.
So maybe to explain that, I'll just give you one last sort of, step of detail in how these synthesis techniques like StyleGAN that I described work, right?
So I'm sure in other aspects of the podcast, you've talked about GANs, generative adversarial networks.
It's a super popular topic. Sure. Indeed, that's what a lot of the most exciting synthesis and graphics is using.
Again, as a review for anybody who doesn't know this already, In our case, it takes two networks and it pairs them up.
One is called a generator, and the generator is the network that we ultimately want.
It takes those random numbers, And it creates an image and then you've got the discriminator. which is, it's often described as an adversary to the generator.
Like the generator is a, an art forger and the discriminator is an art detective, but that's actually, I think it's a bad analogy because really the discriminator's job is to help the generator get better.
So a coach and an athlete is an analogy I've heard that I like better.
So the generator is the athlete. It's trying to learn to make convincing images of, People are cows or bicycles or cars or whatever it is.
And the discriminator is supposed to help it along by figuring out in what way does this not look like a person, a bicycle or a car or whatever.
The way this works today is the generator takes random numbers and it makes a random image.
Initially, it is literally a random image, just a bunch of the big smudge of color.
And the discriminator takes a bunch of real images and a bunch of synthesized images from the generator, and it learns to tell which ones are real, and it gives that feedback back to the generator through gradients.
And this process goes on and on. Gradually, the generator gets better at making something that looks like, let's say, a cat.
And the discriminator gets better at telling the difference between a synthetic picture of a cat or a real picture of a cat.
And this starts because when this starts, the generator is just making a big green mush. a bunch of pixels and the discriminator says that's clearly not a cat, but pretty quickly the discriminator, the generator is making something that looks kind of cat-like and the discriminator has to look for finer details like, oh, the fur texture is not quite right, or the shape of the whiskers isn't quite right.
And, you know, when it's done correctly, These two networks kind of co-evolve.
And when it's all done, the generator has gotten very good at making pictures that can fool a discriminator network. into believing that this is a real image of a cat and not a false image of a cat.
So that's the basic idea of a GAN today.
So what are GANs tomorrow? you know, it seems likely that rather than generating, you know, a random, taking much random numbers and generating an image just from whole cloth out of those random numbers, instead, you're going to have generators that generate a shape.
That might be the shape of a cat or a person or a car.
And another generator perhaps is going to generate materials, right?
A material in computer graphics turn is a description that is a thing that takes Light comes in from one direction and bounces off in another direction, and the material is the thing that describes how does the color of the light change. as it comes in and bounces off or propagates through a material.
So you're going to have a generator that makes shape, a generator that makes material, a generator that sort of makes illumination that models that learns to model the illumination in a real scene.
And so now once you have shape and material and illumination, Those are the basic ingredients of graphics.
And you would render an image from that.
So you need the ability to render an image in a differentiable way.
Right. So you need to be able to propagate from it, you know, again, to getting technical for a second, you need to be able to propagate the gradients from the discriminator where it says, hey, that image is not, doesn't look real.
The sky's too bright. That needs to be able to propagate through this rendering process back to the illumination generator.
And the illumination generator said, oh, I'm not modeling the light transport right.
The sky is too bright. Or the cat's fur is not quite right.
The material generators do that. I think GANs today take random numbers in and make an image, and that is compared to a discriminator, to real images via the discriminator.
GANs tomorrow I think will take random numbers in or perhaps more controlled numbers in and will generate things like shape and material and illumination and animation, the way things move, the way people move.
Those will be the inputs and you will synthesize those things.
And then a differentiable rendering process will make the images which then the discriminator and the videos, which the discriminator then decides are real or not real and guides the generators as to how to make them look more real.
So I think that's the This space occupied by, there's a sort of largely unexplored continent that people are just starting to explore the fringes of kind of in between those two that I described. where you are doing rendering and kind of conventional computer graphics with cameras and material models and light transport and all the rest, but you're doing it in a way that is amenable to differentiation, amenable to modeling with a neural network.
And I think that's really the future of AI and computer graphics.
That's interesting stuff, to say the least.
David, for folks who want to find out more about, and there's so much we've covered, there's NVIDIA research in general, what your group is doing, the stuff that's going on at SIGGRAPH as we're recording this.
Is there a place or places that they can go on the web to kind of dig into a little bit more of the detail of what you've been talking about?
So a good starting spot is the NVIDIA Research webpage, and that's research.nvidia.com.
And in particular, you know, I would go almost straight to the publications because, you know, every paper that we publish, you know, all the preprints that we put up on archive are on the publications page.
And so that's a really good place to sort of browse and see sort of what are we doing in NVIDIA research, not only in graphics research, which I represent, But there's, of course, heavy overlap with vision research, with AI research.
And I think seeing it all in one place like that gives you a sense for sort of the scope and also the trends of what we're doing in NVIDIA research these days.
Excellent. Well, David, maybe next time we speak, and I hope we do get the chance because I'd love to get a... an update down the road on what you and your group are up to next.
Maybe we can do it in high quality augmented reality.
Who knows? Yeah, I'd love that. No, it'd be great to talk to you again.
Absolutely. We'll book it. David, thanks so much.
All the best to you and all your colleagues on your team and in the research group.
It goes without saying. Stay healthy, stay safe, so you can keep creating the future for us.
Thanks so much, Noah. Take care. Thank you.
Thank you.
