Welcome to NVIDIA's AI podcast. Today we are going deep into deep learning. with the head of applied deep learning research at NVIDIA, Brian Catanzaro.
Welcome, Brian. It's good to be here. You've been in the AI game for not as long as AI has been around, but as long as it's been interesting, let's put it that way, or as long as it's worked.
So you've covered a lot of ground from voice recognition to you talk about how You can apply deep learning to chip design, to video games, and clearly there's other applications.
So we want to use this time with you to talk about Where deep learning's been and some of the hard problems that you worked on that you guys were able to solve, you were at Baidu in the past, now you're at back at NVIDIA, but what you see happening today and then what's on the horizon.
So let me back up and ask you first, You make it sound like deep learning is this flexible thing that can be applied to, as you say, everything ranging from chip design to video games. to transportation for that matter.
How is it that it's so flexible? Well, deep learning today is good at finding mappings from an input domain to an output domain.
So kind of like if you imagine like a bunch of X and Y pairs, The strength of deep learning is that it's able to find mappings that generalize pretty well between those pairs. if you have a large data set with a lot of examples.
And it turns out that there is a great number of problems that can be sort of framed in this way where, you have just a huge number of labeled examples and you want a system to learn what that input means.
So, for example, you know, the first success really of deep learning that set the world on fire was image classification and now we're at the point where computers are actually better at recognizing objects and images than the average person is.
And that ranges everything from cats to potentially cancerous spots on a medical image, or clouds from the sky, I guess.
You know, it varies by domain and it also varies, you know, human abilities vary quite a bit, right?
So a radiologist with, you know, many, many years of training is going to be much better at finding cancer in an image than an ordinary person on the street.
So when we say that a deep learning system can exceed human ability.
Usually that doesn't mean it can exceed like the most expert human. but it can exceed sort of the average human or even sometimes an ensemble of humans if you had like a committee of people voting.
In our speech recognition efforts at Baidu, actually, we found that our chinese language speech recognition system in many cases outperformed humans and we would we would have people vote what they thought the correct transcription of a particularly hard section of audio was and we we actually found that our speech recognition system was more accurate Well, let me just ask you about that for a second because when speech recognition gets that good,
What does it start to unlock? And what does it unlock at scale?
What did you see start to happen as it got that great at Baidu?
A lot of these technologies, you can't really rely on them until they achieve that kind of accuracy because otherwise it's just too frustrating.
So I'll give you a humorous example. My wife's name is Jenna.
My boss's name is Jonah. And the other day, I tried to call my wife and it called my boss.
I hung up immediately. And then I tried to do it again because I was kind of stupid.
I was like, oh, it's just voice recognition.
It messed up that one time, but it'll do it right the second time.
They called my boss again. And so then he texted me and he's like, what's going on, Brian?
Is there something wrong? Can I help? I was like, oh, sorry.
It's just speech recognition, you know, kind of running amok.
Um, and so clearly like that changed my behavior, right?
So when I. want to call my wife these days, I do not rely on the voice recognition system on my phone.
I don't trust it. I have to ask though, when you asked your device, call my boss or call my wife, that's not what you did clearly.
You said call Jenna. That's right. I said call Jenna.
It was too close to call Jonah. And, you know, I was in trouble.
That's funny. So it's not... It doesn't help until it's really very, very good and accurate.
That's that's exactly my point. But once you achieve that, then what becomes possible?
When you see some of these systems And some of the problems that they're solving achieve whatever accuracy it's going after.
What does that start to unlock? Well I think it really opens up possibilities for people to do things that we just couldn't conceive of before.
With speech recognition in general, I've spent a lot of time over the past few years working on speech recognition because I feel like the future of human computer interaction, a lot of it is going to be through voice, especially when we start talking about devices that are embedded sort of in our daily lives. feel like having reliable speech recognition can open up a lot of possibilities for example to interact with devices that may not have space for a physical keyboard or or it may just be that when you're using them it's not possible for you to like pull out a phone and poke at it in order to get something to happen.
So you're talking about changing the temperature, opening the door, telling the car where to go. all those things?
Yeah, and more. I think there's going to be a lot of devices like wearable computing, for example.
You know, you can imagine wearing something that monitors your heart rate.
I saw a great biofeedback system the other day that's trying to measure data on how your mood is and how your physical sort of your mood is reflecting physically in your body and like taking data all day.
Well, it's really annoying if, you have to actually pull your phone out of your pocket and like open an app and sign in and like click through stuff in order to input like, okay, this is my current mood.
It'd be much better if I could just say like it could just buzz at me and I could just say, oh, I'm feeling fine or I'm a little bit stressed out.
So I feel like there's that sort of interface.
It's a little awkward to have buttons for a lot of things.
I love to cook. Uh, often my hands are busy and like it just, I don't want to, you know, touch a device while I've got flour all over my hands, for example.
So I feel like speech recognition opens up a lot of new possibilities for human computer interaction, but only if. it's actually good.
Because if it's not good, you might as well just not have it. continuum, we see devices like Alexa and other things out there in the world, you know, Google's got Google Assistant And there's Syria, of course, in this sort of continuum from where we are to where we want to get to or where you would like us to get to.
Where are we? I mean, is this really early days still?
Or, you know, Siri's been around for a while.
Yeah, there's sort of this... effect that happens where As AI gets better and better, it kind of disappears into the background.
And once you stop noticing that it's there because it works so well. that's when it's when it's really landed right I would say speech-driven interfaces aren't there yet.
I'd say we've made a lot of progress. Siri, Alexa, Google Assistant, these are all frankly, way better than I would expect that they would have been just five years ago, like projecting into the future, sort of looking at how the rate of progress had been in AI before deep learning happened.
But I still feel like there's a huge distance to go yet.
Right. And like every good technology just sort of is there and it's not so dramatic and front and center and like, oh, gosh, I'm using technology.
It's just I'm doing what I need. need to do right I don't think we've gotten there yet with with speech for sure but I'm really excited about the future I think speech recognition is one part of human computer interaction, but there's also, I like to think of it as a tripod, the other two parts being natural language understanding.
So kind of being able to figure out the user's intent and kind of connect that into a story because You know, humans are really good at remembering the context.
So you and I are having this conversation.
We're sort of remembering all the things that we've talked about and building on that.
Computers right now aren't very good at that.
So if you want to have a conversation with a computer that lasts more than a few back and forth volleys, It tends to be pretty shallow and unproductive because there's less to draw on in context.
That's interesting, huh? Well, it's it's because I I think anyway because the mechanisms that we currently have for reasoning about user's intent tend to be more one-shot kinds of things like I'll present you with you know, one thing that you said and it'll figure out what that one thing means.
Yeah, it's transactional. Yes. Yeah. But it's much harder like our current systems aren't nearly as good at sort of keeping a memory of and building up a context iteratively and sort of building off that. now there's been some really interesting research along those lines in the past few years and I think we're going to solve it and it's going to be really cool when that happens.
But until that is there, you know, our human computer interactions are going to be limited to these very transactional, like turn on the lights kinds of things where you don't actually need the context that humans actually have when they're interacting.
So I think there's a lot of room to grow there.
And then the third sort of leg of the tripod is sort of the output side, the text to speech side. and if you listen to Google assistant or Alexa or Siri or any of the other great assistants that are out there today and I don't think they've quite achieved the feeling that you get when you're talking to a person, right?
I think everybody would agree there's room to improve there.
And I think that matters. I think these sorts of sort of oh, it just doesn't feel quite right to a technologist, to somebody who lives technology every day, you know, whatever.
I'm just. I'm an early adopter. I'll kind of ignore it.
But to like the larger population and to a lot of applications especially where we're asking computers to deal with people's emotional needs.
I feel like, which I think by the way, that's gonna be very important going forward, is computers that can interact with people on an emotional level.
You can't have those barriers in the way.
It really needs to feel right. And that's because if voice is the interface, We have lots of expectations around voice, whereas if I was just pushing return on something, or clicking a mouse, or tapping a click pad,
I don't have those expectations. Yeah, I saw a researcher once that compared human conversations to flocks of birds.
I don't know if you've ever noticed like, you know, a million birds flying together and they'll all change directions all at the same time.
And And you wonder, how can they do that?
How can they fly in such beautiful synchronized patterns? and it's so instinctive for them.
While humans were kind of clumsy and maybe not as beautiful as those birds, Our ability to carry on conversations, anticipate what the other person is saying, kind of build up a context and provide useful information, exchange information, that's like very sophisticated thing that we do instinctively without even really knowing it and until we get computers to that level the applications for human computer interfaces are going to be limited.
So do you believe then, and it sounds like you do, that there will be kind of deep learning neural nets, DNNs, For emotion.
I know this is kind of a Star Trek trope, right?
Like with Data, like, oh, he just couldn't have any emotions.
Well, and some people I work with, yes. So that's a really interesting topic, but I'm going to move past that for a second.
I feel like building neural networks that can sort of predict human emotions and then predict emotionally appropriate responses, that's definitely within reach of uh current ai technology i think we're going to see a lot more of that and you train the data on what like what is the correct answer and according to whom That's such a great question.
So a lot of applied AI is about getting the right data set.
And it's very hard in a lot of cases to get enough data that's labeled correctly in order to train a neural net.
So the answer to that question, how do you get that data, depends a lot on the application.
And some applications are just way easier to get data for than others. one thing that a lot of people use is crowdsourcing.
So basically going on the internet and asking people or maybe even paying people to give you their interpretation of what's happening.
So for example, you could, you could imagine running an experiment where you had snippets of text or a video clip or an audio clip and you ask people to say Does this clip represent anger or love or happiness or sadness.
How interesting. And so remember I said at the beginning, deep learning is really good at mapping X to Y, right?
So if X could be say an audio clip, Y could be sort of the feeling that's along with that.
That's actually not that difficult of a problem to formulate.
The trick is, do people agree? Right, because we know a stop sign is a stop sign or a cat is a cat.
Is it sad? Is it funny? Is it, I don't know.
So there's a certain ambiguity here. And like, you know, a lot of artwork. is actually built off of that ambiguity, right?
Because humans find these sorts of things really interesting to think about.
Like, is this a sad painting or or a happy painting, there's some paintings that are both at the same time, or a poem.
What does this poem mean? Well, it could mean this or it could mean that, and we can all interpret it differently and talk about it.
So things that are ambiguous like that, I feel like there's an infinite amount of depth where we can get progressively more sophisticated by getting better labels that explain the data.
I can't wait to see all of Shakespeare's sonnets crank through some neural net or You know, name your favorite poet, W.S.
Merwin, whatever it is, you know, and wow, that sounds like fun.
Yeah, but so so like getting a neural net to be as emotionally sophisticated as like a PhD in literature might be very far away kind of task.
Right. But one thing that's been happening for a long time. is sentiment analysis is what it's called in call centers, for example. we all have had the experience where we call in to you know maybe our cell phone service needs to change or something and we have to talk to somebody to to get our account updated and It's very possible with today's technology to monitor those phone calls and get a sense for how a person is feeling.
In particular, the call centers want to know if people are really angry, because if they are, then that's a sign that their customer service isn't working very well.
And so they can kind of measure, like people's emotions based on AI that's kind of listening and understanding how people are feeling that helps them do better on customer service, especially if they notice like this particular kind of problem makes people more angry or this particular Customer service agent could use a little bit more training.
Makes everyone angry. So that kind of like do computers understand emotion, that has actually been happening for a long time. but I think we can get more sophisticated.
There was another thing that I saw recently that I thought was really cool. robotic seal I think it was from Japan and it was trained to kind of comfort people so it's kind of like you know a advanced teddy bear I guess that tries to respond appropriately to people's emotions and It may seem a little corny today, but I think in the future there's going to be a lot of need for that, right?
Because a lot of people feel lonely. And especially as the world's demographics change, there's a lot of people out there that could really use somebody to interact with, somebody that could listen to them and respond appropriately.
I think we're going to see products that do that.
Do you think we're pretty close, at least in some contexts, to this kind of Turing test reality where we're not going to know what's on the other end?
And in many ways, at that point, it sort of doesn't matter as long as I feel better, I guess, or get the answer I want.
I don't know how close we are to solving the Turing test.
It's probably going to be true that we will be able to solve interactions in more constrained scopes first.
And I think that's already happening. And we'll sort of see the scope at which we can solve Turing test kinds of problems expand over time.
But like I said, training a computer to be as sophisticated as a Ph.D. in literature and have like an expert level conversation on like some obscure medieval poet.
You know, like that. That seems farther away to me.
I think that that domain should be reserved for PhD candidates in literature and they should get together in person and discuss it.
Yeah, that's great, right? Yeah, so I think that's a good point is that there's a lot of useful things that a computer can do without needing to do everything.
We described this in the beginning that this is your job, you know, applied deep learning, but that it applies to all these other areas, including chip design and video games.
How does it apply to video games? Oh, well, actually there's a lot of different ways that you can use AI to make video games better and I'm trying to think of the ones that I can actually talk about in public because some of them aren't released yet.
I don't know. Have you ever seen the style transfer stuff floating on the internet?
Like you can take... your face and then a painting from Vincent van Gogh and mash them together, right and So that's deep learning, right, that's doing that.
It knows, generally speaking, where your face is and that it's a face.
Yeah, it's using deep learning to analyze things. the shapes of objects and then sort of it also analyzes the shapes in the image that you're trying to transfer from, and then can kind of combine them in very smart ways, right?
Anything like just like taking pieces of the Vincent van Gogh painting and kind of stapling them on top of each other to make a collage.
It's way more. involved in that it's actually using a neural net to understand both the style as well as the image and then combine them at a higher semantic level and and then sort of project back out into an image.
And so this kind of style transfer work has gotten really popular.
I just saw Mark Zuckerberg put a video on Facebook the other day of his dog that was processed through one of these style transfer things and it looked really cool.
This is an example of how neural networks can be used to generate more beautiful art.
Video games a lot of video games is about art and how do we generate it and so this kind of thing?
I think is gonna really help the industry.
So for example, if we can make tools that will allow artists to be more productive as they're creating virtual worlds, Because right now, a lot of the details inside of a virtual world, like where each tree is or where each building is or Each window, all of those have to be tweaked by a person and it takes huge amounts of time and money.
So imagine if we could train a neural net that could understand this is the structure of what a city looks like.
This is the structure of houses and trees and how they're placed.
And if we could... We could sort of train the neural net to be able to place that.
Then, for example, artists would have to do a lot less work to make more beautiful artwork.
Do you see that going outside the realm video games or art and arts in the real world and sort of video games for that matter but like to planning cities or you know let's say we have traffic issues or whatever it is like how to Think about the variabilities within some system, whether it's a city or a neighborhood or a university campus, say. and bringing AI to bear on that.
Yeah, I mean, it's definitely a technology that can be used for stuff like that.
You would need to have a lot of data. Yeah.
But you could probably get a lot of data from satellite imagery about how cities are actually structured.
I remember when I was in college, they had a lot of green spaces with sidewalks that were set at kind of rectangular angles to each other and so it looked very formal and nice and you would walk like you were supposed to walk on the sidewalks and And instead, there were like these little paths burned through... That would cut off the...
Right, because people needed to go somewhere and they really just didn't want to walk that far.
I bet you could use satellite imagery and deep learning to predict where people are going to want to walk.
And, you know, that's kind of a complicated problem because it's not just the shortest path, but you also want to avoid, like, going up hills or whatever, going around bridges.
And so I could imagine you could use deep learning to do some of that.
I know there's a lot of work using deep learning to analyze satellite imagery of cities to kind of understand where the people are and where the services are. that's very useful to companies that are trying to figure out, for example, where they should expand, right? if you know that this particular neighborhood doesn't have enough target stores in it based on the kinds of people that are living there, then maybe you decide to look more closely at opening a store there.
So there's a lot of analytics problems that are going on where they sort of build up models using deep neural networks to understand how people live. and then use that to predict how well something would do if it was built.
That's cool. I mean, the through line here seems to be whether it's emotions or how humans behave when they walk across a campus or when they shop.
It's this putting human behavior and action in context. and then figuring out the data points to help us analyze that better and do a better job kind of of the outcome or the output.
Right. So the idea is rather than like me trying to sit down and formulate like a theoretical model about humans behave.
With deep learning, I just need to have a lot of data of... actually measuring how people behave, whether that's from a satellite photo or some other way.
Right. And then I can take that data and use deep learning to build that X to Y mapping that kind of predicts, given a new situation, how are the people going to behave?
And that's something that current technology is good at.
It sounds to me like data is the key to much of this, and certainly we have enough cat data, but we're going to need to collect data from all kinds of other... walks of life to apply it.
What are some of the other parts of this that we need to upgrade or rev up if if machine learning and AI is going to imbue all kinds of parts of our lives.
Generally, the world needs to become more data-centric.
I think that... There's a lot of businesses right now that generate hugely valuable data that just don't quite know what to do with it.
And sometimes they might even throw it away.
And so like kind of building a culture of data consciousness that allows you to build up these rich data sets that describe how people behave. we're going to see more of that going on.
And I think I think every company that's thinking about AI and how they could apply AI should be thinking about what's our data strategy and how are we going to retain all of the data that we produce so that in the future when somebody has a bright idea about how to expand our business, the data will be there waiting for somebody to use.
And that X to Y relationship that you described at the beginning, it seems to me that the people who run the business know better than anyone that kind of X to Y relationship, you know, the inputs and the outputs and kind of what they want to tune.
So then they just, how do you, how do you advise them to start thinking about how to bring deep learning into their business? how do they even think about what it can solve for them?
Well, so the way that I think about it is that, so deep learning today is best at those X to Y problems.
And so you need to kind of be convinced in your heart that that X to Y mapping exists.
I see. Right. So speech recognition is a good example.
You and I, if we were listening to this podcast, we could write down the words that were being said and we would more or less agree because that function is fairly well defined.
Even if I don't actually know how to write down that function in terms of math operations, that function that translates sound, samples into characters of text that exists, and I'm pretty confident it exists.
I can implement it. Now, there are other X to Y mappings that I can conceive of that would be very difficult to implement.
For example, an X to Y mapping from... all of the details of a person's life to the novels that they wrote.
Right. So I could I can imagine a, you know, neural network that could write new Dostoevsky books right I mean I
So we don't really have enough data because Dostoevsky only wrote a few books and we don't really have like enough technology to understand what that mapping would be.
And you and I wouldn't agree on it. It's a very ill-posed problem.
So that would probably be a bad candidate.
So that mapping needs to be really clear and we need to be able to collect a lot of data about it.
Another thing that's really important is that the mapping, once you find it, needs to be reusable.
So the process of... actually finding it is very time consuming because you need to get a lot of data, you need to have good computing resources in order to train these models because they're all very computationally intensive.
And so that means that the end product, once you have that model, you should be able to use it lots and lots of times.
And again, speech recognition is a good example of that because once I trained it, then, you know, I trained it generally to handle any kind of speech, then I can use it on anybody who's speaking in it and it works.
Now, if we have a problem where for every different instance of the problem I have to a new network because the way that I framed the problem, the x input domain is different for every instance. then that's not very reusable.
And so it probably doesn't make sense to do that.
So it sounds more subtle than I had imagined that within the corporate world, the business world, There's probably a fair number of those X to Y relationships within a business.
You just need to define the right ones and what can be applied to it.
Yeah. Yeah, another thing that I think of often is sort of whether you can succeed despite failure. any statistical learning system, deep learning is, it's very data driven.
It's going to mess up. The question is, can you provide value to your customers even though you're going to mess up for some fraction of the time?
So that may mean that often deep learning systems are sort of the heart of of some AI product, but it's surrounded by rules that will keep things safe in case going off the rails in case it goes off the rails um can you can so for example um Ad placement, a lot of companies spend a lot of time figuring out AI systems for showing advertisements to people. if they mess up and show you slightly the wrong ad, life goes on, right?
And they can continuously iterate and update their models and just provide more and more relevant content for you.
And even though occasionally they mess up, it's still providing value to them and to you.
So that's that's a good example. Whereas something where, you know, if you're building an emotionally intelligent system and the person needs to be interacting with it.
And in the middle of the conversation, it insults the person or, you know, it could really ruin it, right?
And so you need to sort of plan for failure and plan how you're going to succeed despite having some percentage of the problems not work.
It's cool because we're getting into a very human realm and trying to figure out how to navigate that.
Last question for you. You spent undergraduate work, graduate work, six years getting your PhD, you know, plenty of work, postdoc, I'm sure.
Are we getting enough people who can kind of navigate this world and gather it to themselves and go after it?
Do we have enough people in? in the computer science departments or even in the Russian literature departments who can bring deep learning and AI to bear.
I'm always going to want more people to do this because I mean, I'm, trying to apply deep learning to lots of different things.
And I feel like the demand is really high and the number of qualified people to do it is always lower than than I would like.
So I hope that the education system continues to sort of understand the importance of this technology and emphasize it.
I think there's a lot of encouraging signs along those There's been, you know, huge uptake of computer science education generally that I've heard of in the past couple of years and There's also been a lot of interest in online courses for machine learning.
I just helped out with a Udacity course.
It's a nano degree for self-driving cars.
I'm really excited about that because I feel like self-driving cars are a huge application for AI and there's a lot of demand across many industries actually for people that understand the technologies.
So I think I think there's a lot of opportunities and sort of the education system is kind of adapting.
The one thing that I think is hard is that a lot of skills in AI, the theory is less useful perhaps than the practice.
So we really need to bias people to have a tendency try things out and experiment.
Many things that sound like a good idea before you do them actually turn out to not be a good idea and it's hard to know in advance.
And so giving people that opportunity to actually do AI and get their hands dirty with the data, I think that's That's something that I'm always looking to try to figure out how to do more of that because I think that's really important.
So generally to your question, yes, I think the world is adapting.
I think we're seeing more people that are qualified and can contribute to this space. it's not enough and you know personally as I think about what the future looks like for the the world economy over the next you know many decades I feel like investments in AI are a really good idea because I feel like there's going to be so many applications for this technology across so many different domains that now is the time for people to really be learning and starting to apply this technology.
And that's actually why I'm here at NVIDIA because I'm really excited about the opportunity to do that here.
Just to blow it out and bring it to the world.
Well, you heard it. Get your hands dirty with the data and have some fun and get to work.
Brian, thank you so much. Thank you. PostScript here.
If you are listening to this podcast on iTunes, Google Play Music, whatever your favorite podcast player is, do us a favor and leave us a review.
Thanks for listening.
