Thank you. Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. We're here today with a special guest, Brian Cottanzaro, who is the Vice President of Applied Deep Learning Research here at NVIDIA.
And if you've been following the podcast for a while, you know that his is one of our most popular episodes of all time.
So it's a pleasure to have Brian back. I didn't have the chance to interview him the first time around, so this is extra special for me.
Brian's going to walk us through the latest going on at NVIDIA Research And if we have time, I was told to ask him about a story that involves Andrew Ng and cats.
So we'll, we'll see if we get to that, but Brian, welcome back to the podcast.
Great. It's good to be back. So 200 people, 11 locations worldwide working on machine learning, autonomous vehicles, robots, computer graphics, of course.
Tell us what's going on with NVIDIA Research.
Well, the goal of NVIDIA Research is really to point the way for the company continue making its products better in the future and come up with new ideas that the company maybe hasn't thought of yet that are going to transform the way it does its business.
And NVIDIA Research has a great history of doing that.
Before I went to work at Baidu, I was actually working at NVIDIA Research and I... was working on deep learning back at a time when nobody at NVIDIA really cared about deep learning.
I was able to kind of explain to the company why deep learning was important and why it was going to make a difference to its business because I had been doing that research.
And then that led to the creation of KuDNN and Jensen. jumped on deep learning immediately and started pointing the whole the whole company in the direction to take advantage of that.
So There's a lot of stories like this from NVIDIA research where researchers have been kind of pursuing their own research agenda kind of putting out feelers, trying to figure out what the future might hold.
And then once it becomes obvious to the company that this is something important, then the company just jumps on it.
And that changes the way that we do our work.
It does. It changes the way the world does its work.
Especially lately. So tell us a little bit about what's going on now.
There are two projects that were highlighted on the NVIDIA blog recently, Noise to Noise and some work with Symantec. manipulation.
Yeah. So let's start with Noise2Noise. Yeah.
So this is a really cool project done by our research lab in Sweden and Finland.
And What they figured out, which I think is super insightful, is that you don't always need to have... paired set of images where one of the image has noise in it and the other one doesn't in order to do denoising type problems.
Okay. The standard way of doing denoising is that you train your neural net with a bunch of pairs of images and half of the pairs are going to be the noisy kind of have artifacts on them that you want to remove.
And then the other half is like perfect images.
And so then what you're trying to do is train the neural network to reproduce the perfect images given the noisy ones. and that has tons of different applications.
Like for example, in ray tracing, it's really important because You're never going to get enough rays because it takes a long time to compute all those rays.
So you're always going to have less rays than you want. which means that your resulting ray-traced image is going to have noise in it because without having all those rays, then the image isn't quite converged.
So being able to denoise the output of ray tracing actually makes real-time ray tracing possible.
And so better denoising is really important.
So people have been working on denoising for a while, and the insight that led to this noise-to-noise denoiser is that You don't actually need the clean image in order to do this because as long as you have multiple copies of the same image where or a very similar image where the noise is different then you can train the model on all those noisy images and it will learn to remove the noise nonetheless.
Wow. This is a super interesting insight because there's a lot of applications actually where you don't have access to the clean image.
So for example, in radio astronomy, you're taking a picture of the stars.
There's going to be noise because of the way the camera works.
But you're never going to have a perfectly clean image because it's not possible to build one.
But you can take a lot of noisy images. And then you can train the model on just this huge collection of noisy images to remove the noise.
This is kind of a naive question a little bit, but how do you know that in something like radio astronomy where there is no, you can't see the ideal you're shooting for?
How does the neural net know, you know, how do you know what you're striving for and how are you able to judge accuracy?
Yeah, it's kind of related to just estimation theory in general.
So, you know, the paper, the noise to noise paper actually starts out talking about averaging how if you don't know what the true answer is, you take a bunch of measurements and then you average them together.
You may have done that in your high school chemistry class.
It's a very related idea, just sort of translating it into the domain of neural networks.
And so the idea is that if you get enough samples, the noise cancels itself out because the noise is different, but the truth is the same.
Well said. And so semantic manipulation was the other topic covered in the blog.
Can you speak to that a little bit? Yeah, so this work is really exciting because it opens the door to a lot of new techniques for rendering graphics as well as manipulating images.
And the idea behind it is kind of to invert semantic segmentation.
Semantic segmentation is you take a picture and you want to label parts of the picture, like where are the objects in the picture?
Where are the cars? and color every pixel that belongs to a car, color it blue.
Where's the trees? Where are the roads? And so in semantic segmentation, the input is an image and the output is a map that describes the kind of objects that are present in every pixel.
Okay. So the idea behind this semantic synthesis work with conditional GANs is to invert that.
So the input to the model is actually the segmentation.
That says, what are the objects in the scene?
And then the output is a photorealistic rendering of that scene.
Right. So it turns out we can train a generative model to create photorealistic outputs given only this high-level semantic description of the scene.
This is super cool for a bunch of reasons.
So for example, in image editing, if I was able to change the type of object in an image.
I could do that at a very broad scale, like I could have a huge big paintbrush and just sort of paint trees onto an image.
And where I painted the trees, it knows how to draw trees that fit there.
Right. Right. Or if I wanted to add a car or remove a car.
It's much easier to manipulate the semantic representation of an image because it's so much higher level. all the details about shadows and textures and colors and so forth don't matter because it's just like this is a car rather than this is a collection of little reflections and pieces of a car, you know, so it's a much higher level way of editing an image.
Right. thing I wanted to say about this is that we envision using this for graphics rendering.
So imagine that you had... a graphics engine whose job it was just to produce semantic labels, which look like a computer game from 1982.
Yeah. You know, it's very coarse. You don't need very many colors because there's like 20 different kinds of objects the model knows about.
So you only need to draw. you know, these very coarse outlines of where things are and then that can actually render photorealistic images.
It's a new way of building graphics. And it's exciting because if we can train a model to reproduce a virtual world we can make it much cheaper to create virtual worlds.
Instead of having to manually model textures and lighting, we can just take a video of some environment, run semantic segmentation on it, and then invert that process with this model, And now we have the ability to recreate any scene inside that environment just by changing where the objects are.
So I, of course, being when you said pick up a paintbrush and paint some trees, I immediately imagined my kids making fun of me because I'm such a It's a terrible artist.
But what stage are we at? Because I was kind of leaping ahead to, wow, this would be great for, you know, somebody like me.
Sure. I mean, any size business, but kind of really using some automation and condensing that process of rendering these worlds.
What stage are we at with it? How far away is it from being something that people can use?
Right now, this is a research prototype, and it's trained on the Cityscapes data set, which is from the self-driving car world.
And so it understands 20... semantic categories like trees, cars, roads, street signs, people, because that's what's in the data set.
Right. So our models are great if you're manipulating scenes that have the exact kind of objects that you would expect to see in a street environment. because that's what it's been trained on.
I think the future of this technology, I can imagine it being applied in many different contexts with other kinds of semantic categories.
We open sourced this project actually, and we saw some people using it to do some interesting things.
For example, working on satellite imagery.
You know, you can use this kind of technique to create artificial satellite imagery.
You just need to say this is where the building goes, this is where the road goes, like looking down on the world, right?
And the model can synthesize things just like that.
So I think it's going to have a pretty diverse application in a lot of fields, but at the moment it's a research prototype that's really only been trained on Street Views.
We're talking to Brian Catanzaro. He's the VP of Applied Deep Learning Research at NVIDIA.
He's got teams of people worldwide working on all kinds of amazing stuff that may well shape the future.
Well, I don't need to say that. You guys know why you're here.
We talked about a couple of things, noise to noise, the noise to noise denoiser, I like the way that rolls off the tongue, and the semantic manipulation.
Any other projects that... that are going on that specifically you want to talk about?
Sure. There's a couple of interesting projects that I'm excited about.
One of them is this unsupervised text project problem that we've been working on.
And this is primarily for text classification problems like sentiment analysis.
Like imagine if we could make a model that could Mark a piece of text as whether it's containing like positive sentiment or negative sentiment or classify emotions in a particular piece of text.
That would be super useful for a lot of reasons.
Right. Well, there's this idea from some Alec Radford and some researchers at OpenAI that you can train on unsupervised in an unsupervised fashion on just a huge amount of text and build a character prediction model that can predict that text character by character,
And in so doing, the model has to learn high-level representations of the text that can be useful for real tasks like sentiment analysis.
The way this works is you can imagine that if the model's goal is to figure out what character comes next after having seen a bunch of characters in a piece of text...
It needs to learn how to spell the words, but even beyond that, it needs to learn what kinds of words are being used. so that it can predict what the future words are going to be.
And it turns out that those predictions that it makes about what the text is going to be, given what it's seen in the past, Those are high-level features that you can use to then solve a problem like sentiment analysis.
And it works out really well. So we were pretty excited because we've been able to get this model training on our DGX system in mixed precision floating point using all the power of Volta and And we can do things that in the original paper that got published a few months ago took a month to do.
We can do that in less than a day. And so that really opens up the scope of what we can try.
Sure. And now we're taking this idea of unsupervised text modeling and starting to apply it to some other problems as well.
We're pretty excited about that. Hal, you mentioned you were with NVIDIA Research before, and then you left for a little bit and came back.
So you've been back to the company about two years now?
A year and a half. Year and a half. Okay.
So people recently have been talking a lot about the past two to three years maybe as kind of seeing this explosive growth and everything that falls under AI.
From your own experience, even going back to when you weren't with NVIDIA, but certainly since you've come back, I mean, how much faster are things going?
That rate of acceleration, do you feel it every day?
Absolutely. Yeah, I think everybody working in AI feels the pace of change.
When I first started working on deep learning, I had honestly no idea that it was going to become this important.
And it wasn't actually obvious back then to everybody.
It wasn't obvious to me and I was doing it.
I mean, AI has... is an old technology, right?
People have been working on AI since computers were invented, right?
And deep learning, the core ideas behind deep learning, many of them and have also been around basically since the dawn of time.
Like they're very, very venerable ideas.
So, you know, what, actually enabled things to pick up so rapidly in the past few years, I think it's combination of better data, bigger data sets, better compute resources, and slightly better algorithms that work. more reliably, that's really made a huge difference.
But even to me a few years ago, I wouldn't have predicted that AI was going to be growing this fast.
Nowadays, I think we're just at the beginning because, you know, my job is to apply AI to problems at NVIDIA, make NVIDIA's Products better make the processes, the workflows of NVIDIA better by applying AI.
And I see so many amazing opportunities just waiting for people to pick up. that I think we're going to be finding really interesting, really valuable things to do with AI for quite some time to come.
Any particular areas that you're, you know, either you're keeping an eye on or you would suggest to a young listener out there looking for something to really dive into?
You know, unsupervised learning has been a dream of many people for a long time, and I think we're starting to see that take off.
I mentioned the work on unsupervised text modeling, but you could also make the argument that generative adversarial networks are a way to get to unsupervised learning.
And there's a lot of other stuff that's going on in the field right now where People are attacking this problem of how do we get labels or how do we work without having human created labels for the problems that we're doing?
I'm very excited about that. Yeah. So research is clearly a big part of NVIDIA, NVIDIA's heritage and where NVIDIA is going in the future.
But is there kind of a mantra or a goal that you see as being... the North Star for NVIDIA Research.
Yeah, NVIDIA Research exists to point the company in future directions and set it up for success.
It's interesting over the past few years as NVIDIA has been growing a lot and AI has really taken off, people often ask me, like, was it all a mistake?
Did NVIDIA become so successful making processors for AI sort of by happenstance that it just happened to be in the right place at the right time? and uh you know there's always an element of of fortune in in any success story but i think The part of the story that people sometimes miss out on is that NVIDIA prepared itself for this change by doing the research.
And that's the goal of NVIDIA research is to figure out what things are going to change the future of the company and then build prototypes that show the company how to do that.
And AI is a good example of that. And I happen to be involved in that. as a young researcher at NVIDIA Research, I was the only person working on deep learning and nobody at the company really cared.
In fact, I didn't even know that it was going to change the world the way that it has.
But I was doing the research because it seemed important.
And I can feel that there was something here that might matter to the company in the future.
And then once it turned out to start being interesting to a lot of NVIDIA's customers, we were prepared to address that because we had done the research.
And I think that's the goal of NVIDIA research, and it's been very successful at that.
So obviously you have plenty of things to get back to, but before we let you go, Andrew Ng and the cats.
Yeah. Quick version of the story. Okay, so this actually happened at the Google Brain team, which Andrew was one of the founding members of.
And... They were working on an unsupervised learning research problem, and they built this system called disbelief. that was able to parallelize training to like a thousand servers.
They went ahead and trained a model to reproduce a large image data set And in so doing, they actually found that one of the neurons in this model was selective for images of cats.
And that was really cool because they had never given it a label.
This image contains a They had just asked the model to reconstruct a lot of images, and then it turned out that because there were so many cats in the training set, that one of the neurons ended up being selective for cats.
And so that was exciting because it meant that we actually were learning higher level representations of the data set.
And that's been the goal of machine learning for a long time.
In fact, One of the best deep learning conferences is called the International Conference on Learning Representations, ICLR.
Learning representation, what does that mean?
Why do we have a conference on learning representations?
Well, the idea is that if you can create a model that understands your data at a higher level, then that is the holy grail of machine learning because that has given you power to understand what is actually the meaning of that data, which is the goal behind any AI project.
So if we can just focus on learning those representations rather than having humans try to construct representations of the data, which is how AI had been done up until deep learning really hit its stride. you know, then we can see the huge benefits of deep learning.
And so that's why this, like, premier deep learning conference is called International Conference on Learning Representations.
And I think that the cat neuron that got a lot of attention in the popular press is interesting because it actually pointed out that we were starting to make progress on that.
Right. I'm a dog person myself, but I hear cats are popular on the internet, so.
They sure are. Brian, we'll let you go.
Thank you so much for stopping by, joining the podcast.
So obviously, folks can check out the NVIDIA website.
They know to do that. They want to know what's going on with the company.
Specific to the research team, is there a home for NVIDIA research on the web?
Yeah, there is. It's research.nvidia.com.
Easy enough. Brian, thanks so much. Great.
Thank you.
