Thank you. Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Our guest today is Aaron Lafon.
He is the Senior Director of Real-Time Rendering Research at NVIDIA.
Aaron has led real-time rendering and graphics programming model research teams for over a decade, and has productized many research ideas into games, film rendering, GPU hardware, and of course, GPU APIs.
Aaron's international team of scientists played a key role in founding the field of AI computer graphics.
They were the first to bring AI to real-time computer graphics, They invented key technologies that brought ray tracing to real-time computer graphics and are at the forefront of combining AI and ray tracing. to rapidly increase the realism of real-time computer generated imagery.
In the distant past, Aaron worked in rendering R&D at Pixar Animation Studios, creating interactive rendering tools for film artists.
He was also part of a graphics startup called Neoptica, which created rendering software and programming models for something you might have heard of, the Sony PlayStation 3.
And he also led real-time rendering research at Intel.
Aaron received his PhD in computer science from UC Davis, his MS in computer science from the University of Utah, and also an MS in theoretical chemistry.
Aaron Lafon, thank you so much for making the time to join the NVIDIA AI podcast and welcome.
Thank you, Noah. So there's a lot in that intro.
You've done a lot of things even prior to coming to NVIDIA where, as we're going to get into, you've done and are doing all kinds of stuff.
So why don't we start with the basics? What is it that you do as Senior Director of Real-Time Rendering Research?
Well, you know, the easiest way to explain it is probably an analogy that I point at unclimbed mountains and help the first ascent teams with their risk management and, um, Yeah, I explained that a little bit more, but I am fortunate to have this incredible team of very experienced research scientists who know well how to explore the unknowns.
And I work with them to figure out what are the big problems we want to solve?
What are the things that are impossible today that if we could solve them would move the field?
It's risk management. Doing research is is exploring where you don't have a map and you hit a lot of dead ends.
You know, we just sort of try to navigate that space as productively as possible.
In terms of what the team actually does, though.
That's what I do, but what the team actually does is they're an international research team exploring on this quest to enable computers to generate images of photo-realistic virtual worlds.
These are worlds that humans can interact with. that are fully dynamic.
And concretely, you can imagine architectural design, visualization, medical imaging, computer games, but that are completely photorealistic.
And so there are many brilliant people across NVIDIA that move this field forward.
And as being part of NVIDIA research, It's my job to be taking the biggest risks and looking the farthest ahead in this area.
I like this idea of you as sort of a... a Sherpa in the world of AI trying to lead your team to ascend whatever the equivalent is of Mount Everest in the AI world.
You've been at NVIDIA since 2013. Is that right?
Yeah. Yeah, that's right. And so how big is your team?
Has the team grown since then? How big is it now?
Yeah, it's grown a lot. We are now in five locations in four countries.
Wow. So, you know, depending on collections of postdocs and interns and researchers, we're somewhere, you know, 20, 25 people spread around the world. working together and yeah, the team, the team has grown a lot as well as, uh,
What we do has evolved as a state of the industry moving fast.
Some inventions that we've created, some inventions others at NVIDIA and others have created.
Even the areas that we've looked at have evolved quickly.
AI graphics is certainly one of those places.
Absolutely. So for the uninitiated out there or even the folks who like me who think we know what rendering means, but we've got an expert on the line.
So let's ask you. Can you just kind of explain what what rendering is?
And when we talk about real time rendering research and graphics and graphics, what does that mean?
Sure. In computer graphics, rendering is the process of generating an image from a 3D virtual scene There's many aspects of computer graphics.
There's the questions of how do you create that scene description, and maybe you've seen these making of Pixar films or something, whether it's these wire meshes. that define the objects in the scene.
And there's the description of the material appearance, how light reflects off what color things are.
And there's a description of lights. and then there's the virtual camera as well.
All these things go together and really what rendering is, is a simulation.
It's a simulation of light bouncing through that scene and a simulation of that light reaching the camera.
We are virtual photographers, but more than just photographers, we have to simulate the the virtual world in order to generate those images.
What real-time means is we're trying to do all that in somewhere around a 60th of a second.
And, you know, unlike in If you're making movies, it's very often the case that a single frame of a movie that you see can take hours to generate.
And given that we want to have these worlds be completely dynamic and have humans be able to interact with them uh you know the goal is is trying to get this done in about 16 milliseconds but sometimes faster.
Yeah, I would imagine the gamers out there don't want to wait for the next frame to be generated when they're in the midst of a campaign.
That's not really an option. How does AI play into this?
What is AI graphics and how and when did that get started and why?
Well, to answer your first question, I mentioned that we have something like 250,000 times less time to make an image than you would if you were generating an image for a movie.
And that means we have to find ways to fill in missing information.
We just don't have time to go do this the absolutely correct brute force way.
And so what we saw with AI and deep learning back in 2016, was that it has the ability through the training process to learn how to inject missing information into what we're trying to do, which is generate pictures.
That came about in 2016. Another NVIDIA researcher, one of the researchers in Helsinki, Timo Eila, brought to us these examples of using a deep learning architecture called an autoencoder And he was showing us that he could repair this large damage that he had intentionally done to these photographs and the network could fix it and generate a nice clean image.
And while this was just about repairing damage photographs, we saw the connection that this could be used to fill in the holes and the missing information that we have when we're trying to generate these images in a 60th of a second.
So the first problem we went after with it was the problem of what we call in ray tracing, denoising.
And I don't know if you've ever seen these images that are generated that have the sort of salt and pepper noise.
It's like, if you take a picture with your camera at night and you get that noise that, that, that, That's kind of related, but it's much, much worse when we're generating images.
We sort of very sparsely sample the world And you get this salt and pepper noise that doesn't look like a picture.
And there's this process that goes through called denoising. to generate the final image.
And we thought, well, let's see if we can use machine learning for that.
So that's where this journey started for us almost four years ago now.
Where's the journey taken you since then?
What's going on with your team's work in the field?
And you can kind of gloss through those four years or so to kind of the present day.
But what are you guys working on and how is the field advanced?
That is a very big question. It's one of our words.
So let me figure out where to start with that.
So looking at where we're at now, there's now been multiple deep learning graphics products that have shipped that are making it possible in multiple areas of graphics to give to generate images faster at a higher quality to give artists quicker feedback to generate for gamers to have higher quality images than humans or able to come up with our own algorithms.
And we constantly, one of the things we'll do in my team is that very often, we'll be simultaneously pursuing both a neural and non-neural solution to a problem to really understand where deep learning can have the biggest impact and where some of the best experts in the world will compete with ourselves.
What we're finding is there are certain types of tasks that deep learning can just do better than humans. but it's really the magic is when we combine the expertise together.
We've tried to do these end-to-end learning solutions where we take all of our domain expertise and we throw it out and just replace the neural networks.
And we can make interesting demos, but we can't turn those into real products that impact that really generate value for our customers.
And so it's when we the magic has come from when we've bind our graphics expertise, our domain expertise with neural networks.
And so I think where we're headed is is taking a bunch of steps back is that computer graphics becomes much easier to create the content for computer graphics, the image quality goes up, dramatically, the performance goes up.
And really, you know, I can say where we're at today is that deep learning is a part of the computer graphics rendering pipeline, it's changing the hardware and the software and the end user experience, but we're also just barely getting started.
I think we're still just at the early stages of this journey.
It makes me think of a couple of guests we've had on the show over the past couple of years.
One who sort of if memory serves, kind of stumbled into A way of kind of doing some of the reconstruction of photos like you were talking about your colleague was doing sort of intentionally, but this kind of taking old photos and sort of restoring and colorizing them.
But then another example, even kind of more pertinent, I think, was – I think this was with creating zombie armies, using AI to generate these zombie – armies. within the context of computer graphics, and I think for gaming.
But this person was talking about this notion of, as a creative director or a graphic artist, kind of telling the AI powered system, okay, this is kind of generally what I want to do.
This is sort of what I want it to look like.
Now go ahead and do the grunt work of filling in, making thousands of these things or filling in the textures or doing this sort of background rendering work.
So that, almost like having a, a team of computerized kind of junior level artists to kind of do that work for them so they could kind of go on with the computer vision.
And so it's just interesting to think about as you're talking about the you know, throwing out the domain expertise entirely, you get worse results.
And so that kind of human machine interaction still being so important to taking this kind of work and trying to push it beyond that demo stage.
Yeah, absolutely. To speak to another part of this, I talked about the rendering problem.
There's also this problem of content creation.
How does the virtual world get created? And that's, I think, where we've seen a lot of impact and there's a lot of enormous opportunity as well in addition to rendering. of uh, Looking at some of the work other groups in video research have done, have made it possible to have facial animation that is ends up being much more efficient use of the actor's time they can come in and they can do, um, just a much smaller amount of work and it's much less tedious engineering and even tedious work by artists, which is a creative process.
So we have another example where people, maybe this is what you're mentioning, where you can just roughly paint a picture that looks like this and then the AI system will fill in the rest, but that's just 2D painting.
Eventually we'll get to a point where it becomes much easier for artists to express their intent and have that turned into 3D content without a lot of the tedious steps that are required today.
On the rendering side, I can speak more concretely about some of the products that have been generated from my team's work of the last, four years since we started this as a reflection of some of the impact this has had, and I can talk more about where we're going.
The first that came out of that denoising adventure was the product, which is now NVIDIA's Optics Deep Learning Denoiser.
And this has been widely adopted in the offline rendering industry.
So people that would spend minutes or hours generating a single image can now, the artist can get this almost instantaneous preview from this denoiser's And that's been widely adopted and appreciated.
The next kind of journey after our denoising step was going after the problems of anti-aliasing and super resolution which I don't know if you remember video games from 10, 15 years ago that I had all these jaggies all over the place.
Absolutely. Things are sparkling and pixels are flashing.
Um, And so the process of getting rid of those is called anti-aliasing, and there's a related problem of of taking an image and enlarging it, but making it look like you rendered at the higher resolution Super resolution.
And this is how a lot of times, you know, TV manufacturers are taking an HD 1080p feed and displaying it at 4K in your living room.
But we started exploring that problem after the denoising problem.
And it started as one researcher and eventually went to two researchers for a couple of years, trying to figure out how to make this work.
And when they finally discovered it, that turned into the deep learning super sampling or DLSS product, which was a, an incredibly difficult productization journey for us, but has ultimately ended up with a the second version of this DLSS2 getting reviews.
There was recently a review that stated something along the lines of, uh, if you're not playing with DLSS to turn on, you're missing out on the best possible visual experience.
I remember the exact quote, and this was invented by people who really were state-of-the-art defining the non-neural traditional algorithms, but it was a It was quite a journey in us learning how to build something robust out of this technology, but the end result is something that we just, at this point, we've kind of put our put our arms down, hands down, so we can't beat this with a human-invented algorithm.
It's just that good. That's just where we're getting started is the end of the rendering process in the pipeline.
We're at the tail end of that now in the image processing part.
And if I can dive a little deeper and take a minute to kind of tell you our To tell you where we're going would help if I go a little deeper into how we generate images.
If you're going to tell us where we're going, you can have two minutes, Aaron.
All right. I want to explain that so that I can put it in context where we are now.
This problem of computer graphics rendering.
The simple way to explain it is it starts by simulating the light that would reach a camera. and the detector in a camera.
And so you shoot these rays, these sort of virtual rays from the camera into the scene and they bounce off of the objects in the scene and they seek light.
The goal is to find the lights in the scene.
So you're casting all these rays. And there's a bunch of decisions about – you can imagine light bounces around and, well, which one of those bounces do you trace?
And so there's a bunch of – decisions kind of in this world called sampling of how you choose those radar actions and where they go.
And eventually you end up with this collection of this noisy image which is collection of samples you took and then you start processing it and really that becomes an image processing. pipeline of denoising and all these other effects, motion blur, depth of field, anti-aliasing, super resolution, all these things get added at the end.
So really the things that we've made products out of so far that are at the end of this pipeline where we're doing this image processing.
And if you look at our papers this year across different conferences and including SIGGRAPH, what you'll see is that we're now exploring the use of machine learning throughout this whole process, not just at the tail end, which is where we started.
And it's also where machine learning and deep learning was really well-defined.
Much of this came from computer vision, so it made sense to work on that.
Now we're looking at having machine learning and AI reinforcement learning algorithms, a bunch of different types. actually being involved in that process of choosing where do these rays go?
How do we find light in these images, in these scenes?
And even further upstream. getting into the geometric representation of the scene itself.
If we add neural information to the scene description, can that help the downline process of...
And then feedback loops. One of our papers this year, we had the denoiser, which is, again, this thing at the end of the pipeline running an image space trying to put Humpty Dumpty back together and make an image again, being able to feed information up to the top of the pipe and say, hey, I need more information here.
I need help. And there's another neural network there that receives that information and drives where the sampling happens.
As you're talking, I'm trying to visualize what the whiteboards in your team meeting rooms must look like.
These layers of tracing back through the process and neural nets to get more information to help out.
And it feeds into a question I've been thinking about as I've been listening to. which I'm going to ask after I do my little reset of the podcast.
Our guest today, we're talking with Aaron Lafon.
Aaron, leads real-time rendering research as part of NVIDIA Research.
He has been with NVIDIA Research since 2013 after working at Intel and Pixar and the optical working on Sony PlayStation stuff.
And he's been working on computer graphics for a long time.
So Aaron, you're working in research, you're leading a research team, but you're at Nvidia, the biggest companies in the world working in one of the fastest moving fields in the world, computer technology, computer science, AI, graphics.
So I'm wondering, How do you balance that?
How do you do long-term research working on these things that, as you said, may take years to come to fruition within this super fast moving field or these fields of AI and graphics.
Well, it's a good question. And there's a few facets that come together to make this work.
And part of this is we talk a lot with indigenous product groups. and also our customers and understand how they build their amazing products and also what the fundamental things are that's limiting them from doing even more?
What are the things that we could invent that would let them do something that just seems completely impossible today?
So that's part of it, as I said, kind of identifying which mountains to climb.
But another part of it is that we participate in the academic research community.
And this means publishing peer-reviewed papers. peer review is a great thing that gives us a chance to get feedback from the best academic and non-NVIDIA experts in the world on the work that we're pursuing.
And we also collaborate with people externally as well.
And I think being part of the world research community is important.
And then as we look to pick projects We try to prioritize the ones that could have the highest impact.
Senior Vice President of Research Bill Daley often tells us that we should be able to answer the question that if our research is wildly successful, what will the impact be?
We really want to pick things that are going to move the field if we're successful and that the The risk that we take, the amount of time we take to pursue that should be proportional to the benefit if we're successful.
But failing is part of that. And so another big part of how we do this is we try to fail fast and look for ways to quickly answer the question of, if this idea is a bad one, how can I determine that as quickly as possible?
Because there are lots of ideas out there and finding the good ones is... is the hard part.
The last part of this is, is having a portfolio of projects that span a range of both risk and timeline.
So that in practice, there are some projects in the group that have struck gold and we're working on taking those ideas out to product in partnership with our amazing product groups.
And then there's collections of ideas of projects that are looking much farther ahead and at the early stage.
And I think examples of this include when we started into AI graphics, it was just purely hypothetical that we could combine them in some useful and profitable way.
And it wasn't clear at all. It seemed interesting and exciting that AI could really help graphics And so there was that question of can it and how?
And then there's a question of can it run fast?
When we started working this technology, it was about a thousand times too slow to work in real-time graphics.
And at the end now, we're shipping products that But it's a case where we had the fortune to be part of starting this whole new field of AI graphics or being part of the early people that were working in this.
And yet at the beginning, it wasn't clear at all.
But it was one of these high-risk, high-reward endeavors.
Yeah. So kind of looking ahead, and obviously there are things that you're not at liberty to talk about, so answer this question as you best see fit.
But what are you and your team working on?
What what challenges maybe, you know, kind of more on a high level?
Do you and your team and then just the field as a whole of AI graphics, What are you facing over the next couple of years, five years, whatever the right timeframe is?
And then I kind of and this is sort of an aside, but wondering what it's been like to shift, you know, during the pandemic to shift from. to working from home and what challenges that's brought about, but maybe even what, you know, kind of,
If anything, what new ideas for workflow and working on these big problems as a team that that might have brought up?
Well, those are two very different questions.
Actually, I'll answer your second one first.
In some ways, our group was very prepared, was quite prepared for the work from home in that we are a distributed team.
My group is in four different countries.
And so in that sense, we were sort of ready for this, but we do have clusters of people who work together.
And what I've found is that If we have the idea, we can pursue that idea in a distributed fashion pretty well.
The hard part had always been, how do you come up with those ideas?
So many of our new ideas came from people that were in the same room.
The execution of the ideas can happen in a distributed fashion.
Creating them is much harder. And identifying which of these peaks are we going to go climb?
That's the part that's hard to do remotely.
And we're getting a lot better at it. Part of what we're doing is creating these blank meetings where we come in with no agenda other than to talk about ideas.
Nice. And just the sound of silence can actually take us into whole new directions.
But it is, I'd say that's the hardest part of it.
Yeah. And so then. the first very different question and sorry for lumping those two together at you, but the, um, the challenges and, and what you guys are working on and where the field's headed.
The, um, I think challenges to the, well, let me start with going back to some of the opportunity.
I think what we've seen from both the products we've created and the many, many more prototypes that we've had in the papers we've written and that others in the field have as well is there's tremendous potential here.
Deep learning can inject missing information into the rendering process.
We don't have anywhere else the idea of coupling all this together. as well as being able to run renderers backwards.
Differentiable rendering means being able to take an image and turn that into geometry.
This has been a long-term Holy grail, the computer vision field, computer vision, AI, and graphics are starting to work together to develop that technology.
There's tremendous potential for where this can go.
I think one of the biggest challenges that we're facing as a field in machine learning graphics specifically, is in this notion of generalization, of taking...
You know, machine learning is this funny thing where it's very easy to get an early prototype that does something amazing.
I call these deep learning circus stunts because they're not actually useful.
They're fun to look at, but you can't actually do anything with them because they don't generalize.
All you've done is memorize the answer. taking these prototypes and turning them into algorithms that are robust and generalize across input it's never seen before, is a huge challenge in our field.
It relates to how we generate the data that we train on, what data we train on, when we train, how we train.
And even metrics, some of the work my group is pursuing right now, a paper this year on this as well, is new metrics for assessing is an image good or not?
Is an image better than this one? Well, what does it mean?
Because when we hand off to a training mechanism, a deep learning training run, the human's no longer in the loop.
The machine is deciding what's better than something else.
And The way it decides that are the metrics that we give it.
And so we're designing deep down for that, but it's This generalization problem and related to that is extrapolation.
Many of these networks are capable of sort of interpolating between inputs it's seen before, but not extrapolating the things it's never seen before.
And I think that's a fundamental challenge in the field.
And really it's about data. It's just, you know, we can As I said, it's one thing to prototype something, but it's another to scale out and say, What is the data we need to capture?
How do we get it? Which data do we use to train?
How do we create a generalized model? One of the things that computer graphics, unlike computer vision, If you can't simulate your data, you have to go capture it in the real world, which is laborious.
The great thing about computer graphics is we can generate We can generate petabytes of images, but you have to generate the right images.
So I think this whole exploration of can we automate the generation of the training data how do we build these generalized models and build generalized models that can work together, is key to making this more practical. and taking many of these prototypes and really having impact with them.
So I think that's probably our biggest challenge.
Aaron, you mentioned a couple times some of the papers that your team has published or is working on.
For folks listening, who might want to dig into the research papers or even at a higher level, just kind of see what you and your team have been up to.
Are there places online that would be good starting points for them to check out?
A good starting place would be NVIDIA Research's website.
All of our publications are there. Of course, each of the conferences we publish and their digital libraries also host that.
Right. I think NVIDIA Research's landing page is a good place to start there.
And that will include research from all of NVIDIA Research, not just my team.
Perfect. And then we'll see in between the time we're talking and when this gets published, we'll see if we can get some specific links together to put in the show notes as well.
So Aaron, this year, as with many, many things in the world during the pandemic and the shelter in place and whatnot, SIGGRAPH is happening, the annual conference, but it's a little bit different this year.
And so as we're recording this, on August 18th, I guess there's some content available from SIGGRAPH, but then next week, which to screw with everybody's concept of time a little bit more. is when we're planning to have this podcast go live.
So as you're listening to this dear listener, most likely SIGGRAPH is happening as a virtual conference right now. with all the content out and some live Q&As and that kind of good stuff.
But the point is that your team has a few papers that you're publishing at SIGGRAPH this year.
So I know you're excited about that. So you want to talk about that a little bit.
Sure. NVIDIA has a number of papers and courses this year.
To talk specifically about two of these, I'll start with one which is directly related.
I mentioned earlier and directly led to this AI and rendering discussion and that is, This paper where we are learning a neural representation of this 3D scene description, and then replacing whole parts of the graphics pipeline with a neural algorithm.
More specifically, we'll learn this neural scene representation and then use a traditional rendering process, non-neural, to cast those rays initially into the scene to hit the first thing, and then all the lighting, all the bouncing they would have otherwise done We don't actually do.
We replace that with a neural network. Or we have a different version of this where maybe we'll compute the shadows that hit those points, but then all the indirect light that comes from bouncing around the room. will be replaced by a neural network.
So it's, again, it's exploring further upstream the up the pipeline from just the post-processing where we started to deeply integrate neural rendering. all the way up to the top of the pipeline as well as the middle.
And similarly, we have a paper that will be coming out later in the year at SIGGRAPH Asia at the end of the year where we have the talks about the neural network, really get involved in choosing the direction that those rays bounce to sort of seek light more efficiently, to pick smarter radar actions.
The other one paper we have this week at SIGGRAPH does not use AI, but it is a significant advancement in real-time rendering in that, It's about being able to render, generate images in scenes that we just couldn't even fathom a couple years ago being able to handle and that is imagine for example a night city scene with all the lights in the skyscraper and the streetlights and the cars and it's all those lights are casting shadows and all of that can be dynamic.
Normally the way this is done in games is much of that lighting would be pre-computed.
It would be static or things would glow, but not actually cast shadows.
This approach lets us have all the emitters that an artist could want or put into a game.
So emitters just being something that is casting light, can cast dynamic shadows.
Whereas the state of the art, even I would say today in games is that a handful, maybe five, maybe 10 lights will be allowed to cast dynamic shadows.
And this takes it from Wow. Ten to millions.
And it's it's a partnership we've had with Dartmouth that has produced this pretty big breakthrough.
And it's, you know, taking a big step back, just there's, it's a really fun time to work in this field because you have the combination of a couple of years ago, the introduction of ray tracing hardware, which allowed us to render the way that I've described now, but in real time, and combined with the Tentacore, hardware that's in our chips and using those together, have a chance to invent algorithms that can combine these together It's kind of a once in a career opportunity to have the pace in the field is really accelerating quickly.
Yeah, no, that's amazing. And that just that notion, the way you described it, I mean, going from, you know, five dynamic emitters to millions.
I think anybody can grasp like that. That's an incredible moment.
So congratulations on it. Best of luck presenting at the two conferences, virtual as they are this year.
Thank you again, Aaron, for taking the time to come on the podcast.
And we look forward to following the progress and, you know, maybe catching up again next year sometime.
Thanks for having me, Noah. Thank you. Thank you.
Thank you.
