Welcome to NVIDIA's AI podcast. We are super excited to have with us today one of the pioneers of deep learning, one of the forces in artificial intelligence, Andrew Ng.
Andrew, Welcome. Hi. Thanks, Michael. Great to be here.
We're in Palo Alto and kind of off the 101 somewhere here and so deep in Silicon Valley.
And of course, all of you know Andrew was at Google.
He's an adjunct professor at Stanford. Baidu, you're the co-founder and chairman of Coursera, correct?
Co-founder and co-chairman. Co-founder and co-chairman, you have to be busy enough.
And has founded a new company, deeplearning.ai, which maybe we'll hear about, maybe we won't later.
Andrew, I want to start with the statement that you've made that AI is the new electricity.
And what do you mean by that? I think about a hundred years ago, as we started to roll electricity in the United States and other countries, electricity transformed every major industry, everything ranging from agriculture, which was transformed through the rise of refrigeration, to transportation, communications, healthcare, they are all deeply transformed by electricity.
And I think now we see a surprisingly clear path for AI to also similarly transform every major industry.
One thing I look forward to is I think your listeners are listening to this on what may be an electric powered phone or electric powered radio, but we don't even use that term anymore.
I actually look forward to the day where our self-driving car is no longer an AI-powered self-driving car, but it's just a car.
Electricity is a utility. And in the same way, AI kind of disappears and becomes a utility that's just there all the time.
I think AI will transform a lot of industries and one of the signs of when a technology is really successful is when we no longer think about it.
In fact, AI is already transforming all our lives.
Every time you open up your e-mail, your spam filter saves you from having to wait through tons of e-mails without spam filters, the email system today just wouldn't work.
But it works so well that you don't think about it anymore.
Every time you use your credit cards, priority learning algorithm checking if it's you or if someone might have stolen your credit card.
And every time you go to a lot of shopping websites recommending products to you, AI Today is very lucrative with online ad placements.
So AI is already a pervasive part of a lot of our lives and it's interesting how the successful examples keep disappearing into the background.
I hope this will only accelerate in the next few years.
You say that there's a clear path toward AI becoming this ubiquitous thing.
It might be clear for you. It's not so clear for probably most of us or the rest of us.
What is that path and when did you start to see it?
Because I know you've been working on deep learning in particular and AI more broadly for some time.
I was fortunate both at Stanford as well as in my role at Baidu to see AI intersecting with a lot of other industries.
So in both roles really, we've had leaders of multiple industries ranging from construction, to retail, to manufacturing, to logistics, come to us and share with us many details of the applications they're working on.
I think that for a lot of vertical industries, there is already very clear value that AI can create to transform these industries.
I don't know, take retail, today we have people making buying decisions where you look look at the catalogs and say, oh, I think purple shirts are in season next year.
And then you just buy 10,000 purple shirts and hope it sells.
So I think that relatively simple AI could help make some of those decisions much more thoughtful and systematic.
Healthcare, just a few days ago, my students published a paper in collaborative iRhythm showing that AI deep learning can read ECG readings very accurately or diagnose a level comparable, maybe superior to a single cardiologist.
There's still a lot of work to do to put these types of AI into the workflow of healthcare systems.
But I think that the path is becoming surprisingly clear, even though there's still a ton of work to do.
Now, in that path, and let's talk about purple shirts and EKGs, but what are the kind of similarities?
What is... this path and what is AI so good at that it's, again, like you say, making this path clear toward it seeping into more and more industries.
Despite all the hype about AI, I think almost all the economic value created by AI today is through one idea called supervised learning. which means learning input to output mappings, such as input and email, that's the input, tell me if it's spam or not, that's the output.
Or more sophisticated, input an English sentence and output the French sentence, the French translation. or input an audio clip, maybe an audio clip of us speaking on this podcast and output the text transcript.
So that's speech recognition. So, it turns out that there are challenges to taking this idea of learning input-output mappings and fitting them into a bigger business process or workflow. information about what people bought in terms of fashion over the last few years.
And I'll put what's the estimated demand for purple shoes next We are very valuable.
So I think a lot of the work to be done in AI is taking supervised learning, as well as other forms of AI that are still being invented and work on and fitting them into the appropriate business context.
What are some of those other forms of AI that people are working on that get your interest piqued?
Other than supervised learning, I see transfer learning getting a lot of traction.
The traction is getting is much larger than the PR you see around it.
But this idea that you might have, say, a neural network learn to do one thing such as recognize images, and then take what you've learned and apply it to a totally different task, such as to read radiology images.
Or maybe you have a deep learning algorithm, learn to recognize Mandarin Chinese speech and then use what you've learned to accelerate your ability to recognize a specific dialect of Chinese such as Cantonese.
So I think other than transfer learning, there are a few other forms of learning that are just kind of on the horizon.
I think that GAN's generative adversarial networks is getting a lot of traction.
I guess Ian Goodfellow, my former student actually, has spoken on this podcast just recently.
Yes, he was great. He was adversarial, I have to say.
Actually, one thing that's not widely known is early history of the rise of GPUs and CUDA applied to deep learning. was at Stanford and actually helped out with that work way back in 2008, I guess, when this was just starting to take off.
And then other than that, there's a lot of buzz about reinforcement learning, I think There's a lot of excitement about that.
It's great for playing video games. It's also getting real traction in some robotics applications.
But I think there's still the real application of reinforcement learning is still in the early phases of being developed, I think.
It's interesting to me because what you describe, this difference between supervised learning, A to B, versus these other kinds of learning that you describe is this kind of next phase for AI.
But if you think about it more broadly, then what does learning mean in the the direction that you and others are taking AI and learning on smart systems, what does that mean and how does it happen and how does it work?
Yeah, machine learning is this slightly mysterious thing.
I think Carnegie Mellon University's Tom Mitchell gave one of the clearest definitions of what is machine learning.
So a slightly simplified version of Tom Mitchell's definition is that if a computer goes through an experience such as examining a lot of spam email data.
And through that experience, learns to do a task better, such as figure out, recognize what is spam and what isn't spam email.
Then that is machine learning. I think we see a lot of examples today, such as a machine examining a lot of data in English-French pairs of sentences and through that experience learns to become better at translating English into French.
But I think the broad general definition of learning due to Tom Mitchell is when a computer through some experience, such as examine a lot of data, becomes better at a specific task, usually as evaluated on some metrics, such as how accurate is the translation.
Let me ask you about your past in this, and how you came to it. always fascinated by AI and machine learning or did you just stumble into it somehow along your education or in practice?
I was fortunate that growing up in Hong Kong and Singapore, my father, who's a doctor, was interested in applying AI to healthcare.
Something like many decades ago, my father was actually writing research papers on using what we now consider very old AI technology to help with automated diagnosis.
So when I was a kid, maybe about 15 years old, my father actually started to have me read books on neural networks.
Oh, wow. And then I did a summer internship. when I was in high school with National University of Singapore and started coding neural networks.
I was always fascinated at the opportunities for AI to help us automate tasks.
Maybe one other story. Once in high school, I once had a job in an office.
I wish I was the office assistant. I think I was actually the office assistant's assistant.
So I remember doing a lot of photocopying and the highlight of my summer was I got to use this shredder.
That was exciting. So I just remember doing all this photocopying and thinking, man, I wish we could write software to automate these things so that maybe we humans could spend our time doing more meaningful, more challenging things.
Well, I'm going to ask this seriously, because as parents, I guess we should all be giving our kids things to read about AI.
And so I'm going to ask you a serious question as the co-founder and co-chair of Coursera.
How can people who just want a... steer their kids towards something or steer themselves towards something, how can they learn about it and what are resources out there that you would suggest?
That's a great question. I think that what's happening with AI is that whereas, let's say five years ago, there was a certain set of things we knew how to automate with computers, With the rise of modern deep learning AI capabilities, the set of things we know how to automate is suddenly much bigger than a few years ago.
So this will create challenges in the labor market in jobs.
And for better or worse, AI will displace many jobs.
So I think we now live in a world where lifelong learning is no longer optional.
The old model of education where you go to college and then coast for the next 40 years of your career, that's not going to work.
And all of us have to keep learning certainly every year, maybe even every week for the rest of our lives if we want to have great careers.
So to parents, I would say, Teach your kids to be lifelong learners.
That's probably the single most important skill they need in order to have great careers.
That's great advice, and I think as somebody who, we work and live in Silicon Valley, Like you say, you have to kind of keep moving forward.
Sounds like more fun to learn something new every week or every month or every year or so. as opposed to just, oh, I got it all, and now, like you say, I'm going to coast for the next 40 years.
Yeah, you know, there's an interesting...
I think the New York Times article just a few weeks ago showing what a cohort of unemployed individuals were doing, and one thing that really intellectually wasn't surprised by but emotionally surprised me by was how little time people spend learning new things.
And so, to your listeners, this might be actually a good sign because on average, humanity doesn't spend nearly as much time learning new things as I think they should.
But what this means for individuals is that with some effort, you could actually get ahead in your career a lot.
Right, right. Well, and that effort, sort of the effort required is, or at least the access is much more available now than it's ever been.
So I think the opportunity to learn is certainly there.
Yeah, I think with MOOCs like Coursera but also many others and so much free content on the internet, It does take hard work.
One of the challenges of learning is that if you spend all weekend studying, maybe taking an online course, Your boss on Monday doesn't know how hard you worked all weekend and you gave your entire weekend to study.
And realistically, you're actually not that much better at your job on Monday just because you work all Saturday or maybe all Saturday and Sunday.
So there are almost no short term rewards to working really hard.
But the secret to this is that it's not about working hard one weekend.
It's about doing it weekend after weekend. maybe for a year.
And if you do that, then after a year or maybe after three or six months or something, you can actually gain new skills and be much better.
But we know that learning does work better when it's spaced out, when you keep doing it for an extended period of time.
So it's really a marathon, not a sprint.
The secret is out now. And so we will all be studying for the next three, six months to a year and get things done.
Next 40 years, actually. That's true. And never stops.
Don't forget that. We were just talking about Ian Goodfell a little bit.
And obviously he was a student of yours and you were there at the beginning of. of applying or leveraging the GPU for machine learning and AI, What was that like and how did you guys come across that as an idea and then execute on it?
You know, I remember it was back around 2007, 2008.
I was at Stanford at the time. We started to hear signals that this GPU CUDA thing was really taking off.
And to give Ian Buck credit, he was thinking about deep learning even in the very early days.
And I think it was, really cool there that flipped the switch from deep learning on GPUs being a very difficult thing to do to something that we should all jump onto.
So remember, it was actually at the NIPS 2008 conference that one of my students, Richard Raynor, and I had published a workshop paper on applying GPU on CUDA to deep learning.
I remember talking to some of the other senior figures in deep learning at the time like jeff hinton and a bunch of others and they're actually very skeptical initially but and do you remember why like were they just like ah well Yes, it was that GPU programming was really difficult.
So there were earlier efforts, but it was just very difficult to scale because it's so difficult to program.
I was sitting down with them and saying, look, the school, the thing is new.
It makes it much easier to program. And I think that convinced several of them.
Then subsequently, Jeff Hinton and Alex Krzyzewski did this wonderful work on the AlexNet and But I think NIPS 2008 was when it really started to come into the deep learning leaders' awareness that cooler is different.
And again, I know this is a video podcast, but I've said elsewhere before, Jensen Huang is one of the CEOs I genuinely deeply admire.
He was early to deep learning, and he and Bill Dally, Brian Casanzaro, It's a work of Adam Coates and me to develop some of the HPC systems using GPUs Jensen, to give him credit, really saw the trend a lot earlier than a lot of people and made a bold bet that turned out to be a fantastic Oh, yeah.
I remember I was a journalist covering this back then.
I couldn't see it, but Jensen was talking about it.
He's like, oh, no, no, GPU is going to move to the center of computing.
I find that one of the things about innovators is sometimes when you see the signals that something's going to work, You just have to make the leap and you don't build consensus.
When you have a conviction, just go for it.
I find that I do that myself too, actually, and sometimes it's disconcerting. to some of the people I work with that I'm making a leap that feels a little bit early, but it's worked out sometimes.
And really Jensen made that leap very early for GPUs.
Well, so one of the things that made us all believe that actually deep learning could work, because I know it was something that had been thought about and written about and just wasn't very good, was your work... with cats and image recognition.
So how did that happen? Yeah, that was a strange one.
So when I was leading the Google Brain team, we were actually using CPUs at the time.
But when I was leading the Google Brain team, we trained on a lot of unlabeled YouTube images.
My student Kwok Le that I brought into the Guru Brain team started to say, well, what appears a lot in YouTube? and maybe we could see if the neural network learned it.
So we thought human faces appear a lot in YouTube videos, so we found the neural network could learn unsupervised learning. to detect human faces.
And it kind of almost is a, you know, not quite a joke.
It was like, all right, there are a lot of cats in YouTube videos.
Let's see if a neural network had learned to do that.
And I still remember when Kwok walked over to my desk and said, wow, I found this cat neuron.
I was like, okay, that's great. So I think Katz, through that paper, has for some strange reason had a disproportionately large mindshare in the world of deep learning that I'm not quite sure is entirely justified but you know it is what it is and i blame you for it um and as a dog owner i think that dogs should be given their due at some point too I'm with you.
I'm a dog person, not a cat person, but I think we're stuck with cats.
All right. Fair enough. Pardon the interruption, but we have a favor to ask.
Leave us a review on iTunes, Google Play Music or whatever your favorite podcast platform of choice is.
It helps more people find us. As always, thanks for listening.
Now, back to the good stuff. Let's back up to 2008 when a lot of this early stuff on deep learning was happening and happening thanks to you.
How did you... I don't know, what was it like then and how did you kind of understand that you were on to something there and what was your evidence?
I think that GPUs for deep learning was a controversial thing back in 2008.
But I think that even back in 2008, my Stanford students and I felt very confident that this was going to take off.
I think one of the differences between what machine learning insiders versus people they join later see is that as a machine learning insider, I feel like I don't actually feel this abrupt shift in machine learning that I think machine learning outside the sea, because when you're an insider, what you see is maybe machine learning is getting 50% better or some number. every year, year on year.
And it's been like that for the last 15 or 20 years.
So as a machine learning insider, I just see this constant But to an outsider, exponential growth often looks like it came out of nowhere.
Right, right. It's this Cambrian explosion all of a sudden for us.
And like you say, you've mentioned it twice.
There's PR, there's hype. You've used those words.
Is it hype, or is there hype, I guess is the better question.
I think there is hype, but AI is also creating tremendous value.
One of the challenges of explaining the value of AI is I think we have our sights set extremely high or transform every major industry, but it's not so high as that we have a clear path for developing sentient AI.
So, I think one of the challenges from a public communications perspective is explaining How come our sights are set so high that whatever industry you're working on will probably transform it in AI? but not so high that you need to worry about sentient evil killer robots walking up to your front door anytime soon.
Right. Well, let's talk about then if AI is going into every industry, and we all work by definition, I guess, in every industry, or at least some of us do.
How do I if we talked about learning how the machines learn?
How do I as a person in industry learn about AI and how do I then sort of learn how to integrate it into my work and into my industry for that matter and kind of have a better understanding of what it can do and then how I relate to it.
I think that business leaders should pay attention to AI.
Just as today, I see a lot of S&P 500 CEOs that wish they had started thinking about the internet strategy or the mobile strategy five years earlier.
I think that in the future we'll see a lot of leaders that will wish they start to think about their AI strategy earlier.
I think that today, the technical knowledge about how to build certain types of AI systems is becoming more widespread.
A lot of people started off with my Coursera course to get involved in machine learning.
There are a lot of great resources out there, including NVIDIA's website and a lot of other I think the skill set to take recently invented AI technologies and fit them into a product or business context, that skill set is still very rare.
But if you're able to hire a good AI leader or build up an organization with a broad-based capability in implementing AI machine learning algorithms. maybe we'll slowly figure it out.
To make another analogy, with the rise of the internet, maybe like 20 years ago, it took us a long time to figure out how to build truly Internet companies.
Maybe here's an example. I have a retailer friend and his CIO told him several years ago that, well, we built So we're just like Amazon.
And the reality is, no, this is not just like Amazon.
You can take a shopping mall and build a website for the shopping mall.
This does not turn your shopping mall into an internet company.
So what defines an internet company is not whether you run a website, is whether you've built your company, such as letting engineers and product managers, rather than the or just a CEO make a lot of key decisions or whether or not you do A-B testing pervasively, and have short iteration cycles and learn from data.
Those are the things that really define an internet company.
And truly internet companies had to build their organizations this way in order to leverage new internet capabilities, such as you can run a ton more experiments when you have a website.
That's what defines an internet company.
I think we're still in the early phases of figuring out what it really means to be an AI company and how to build your organization to leverage AI capabilities.
20 years ago, none of us, I certainly didn't know that A-B testing was going to be such a big thing.
It took us a long time to figure that out.
And I think today we're still in the early phases of figuring out what are the things that we will look back on and say, wow, these were really important for building a company to leverage AI capabilities.
Do you have any early indicators of what some of those things are?
And also, we talk about how the sort of AI literate or not just literate but expert are sort of a rare bird.
I mean, there's not a lot of people like you out there.
Are we going to have more people who are experts in AI?
What are some of those early... signs of ah this is what those native ai companies are going to look like or what they're going to need Honestly, I don't think I'm that special.
I think there are tons of fantastic AI researchers. but all of us still just have a lot of work to do.
In terms of early signs of how to be an effective AI
I think that companies like Google and Baidu as well as Nvidia are among the leaders in terms of figuring this out.
I don't think any of us have the right answers yet, but I find that AI companies tend to be much more strategic in how we think about the use of data.
So for example, I've done things like launch a product in one geography, plan to use that to generate data to then launch an adjacent product in this adjacent geography.
But then we don't plan to make money off any of that.
We plan to take this data and monetize it in a totally different location.
So, we're starting to understand how to play these, frankly, sometimes they're multi-year chess games in terms of very sophisticated data acquisition strategies.
AI companies tend to have centralized data warehouses.
Traditionally, we saw a lot of internet companies with distributed data warehouses, by me, logically distributed, not physically distributed, but by putting data together, that's when you can enable your engineers as well as your products, examine enough data that often spans multiple products in order to create the most valuable services.
I think that we're also seeing the traditional job descriptions for internet companies break down in the AI era.
For example, if you pull up any mobile app on your cell phone, chances are there's a product manager that drew a picture called the wireframe that helped an engineer implement what your app should look like.
But if you take self-driving cars, it doesn't make sense to wireframe a self-driving car.
That's not what's important about a self-driving car.
So, leading self-driving car organizations like Drive.ai, where I'm on the board, is inventing brand new processes for product managers. to help figure out what the product needs to look like and how to communicate to that engineer.
So I'm finding the old job descriptions of what an engineer, product manager, designer, need to do, those old job descriptions are breaking down and we're starting to invent new job descriptions and processes for these things in the AI era.
So, I mean, it sounds to me that as you described what turns out an internet company was, Amazon, not a mall with a website.
The idea that internet companies can necessarily or naturally just evolve into AI companies, that may not necessarily be true either.
With the rise of the internet, there were many new entrants such as Amazon, Google, Facebook, Baidu, and so on. were able to create a ton of value.
I think because they architected their companies to leverage brand new internet capabilities that didn't exist before.
I do see these companies as trying to move rapidly to re-architect themselves to take advantage of AI capabilities.
But I think that just as with the mobile revolution, it was actually Uber and Lyft and Airbnb that figured out all the things about... how to do things on mobile that the traditional internet companies did not.
I think that there'll be brand new opportunities as well for new entrants.
You mentioned product and how products are going to be developed differently, but there's an argument to be made that most products in some part are software-based.
But is there any kind of industry out there where I can be like, oh, well, this all sounds fascinating, but AI is not going to impact me.
I'm just going to do the things the way I have always done them.
Some of my friends have this game where we challenge each other to name an industry that we don't think will be impacted by AI over the next several years.
And I actually have a hard time thinking of one.
My favorite example was my hairdresser. Your hair looks great, by the way.
Oh, thank you. I don't know how to build a robot to automate that job.
Having said that, I once said this on stage and one of my friends who's a robotics professor was in the audience.
And after she came up to me and she actually pointed at my head and she said, Andrew, for most people's hairstyles, I don't know how to build a robot to automate their haircuts, but for your hairstyle, Andrew, I could make a robot do that.
Ooh! Well, so there you go. Hairdressers, at least for you, might be roboticized.
It is true, though, that it's hard to find an industry that won't be touched by AI.
I've been asked by a lot of people, what can AI do?
One of the rules of thumb I've given is that almost anything that a typical human can do with less than one second of mental thought, we could probably now or in the near future automate.
This is an imperfect rule of thumb, but things like looking at a picture and telling me, you know, There are pictures of your friends in this picture.
A human can do that in less than a second and the AI is getting really good at it.
And so a lot of the challenge in rolling out industry and industries is figuring out where to fit these one second capabilities into a bigger business process, into workflow or product.
So take maybe a job of a security guard, right?
A lot of the work of a security guard monitoring video feeds can be broken down into a string of one-second toss, such as watch a video, see where the people are in it, and then not seeing the people, see if any of them are doing something suspicious.
That could be right for automation, but it takes a lot of insight into the job of security guard or the job of call center operator or job of a radiologist to see which parts of these things are right for automation.
If I'm a software engineer or a coder, how do I prepare for AI?
Some folks who talk about how well software is going to write software, and that's already happening to a certain extent, and I know that Google announced that it wants to make deep neural nets far easier and far more automated.
As an engineer, how do I prepare for this AI future?
Or present and future, I should say. You know, one of the things I love about working in tech is that tech advances so rapidly.
So the ground is constantly shifting under our feet and there's always something new to learn.
We're actually used to, in tech, all of us just have to keep learning.
You know, I think that they're onto there isn't nearly as much Fortran programming work today as there used to be.
And we're just used to the barriers, the boundaries of what's possible shifting and then all of us just shift with it.
I think that there has been a bit of excessive hype about software writing software.
We've actually had this for a long time.
Even the PostScript language or PDF files, that was actually a programming language.
It's just a computer wrote a piece of code that told your printer how to print a document.
So that was software writing software. And of course, this will accelerate.
I mean, if you take iOS, I think Swift actually has already made developing mobile apps much better and easier.
So software tools will definitely keep advancing.
But there are parts of programming that I don't see a clear path to having AI automate.
So I think that software engineering jobs are actually relatively safe from massive automation. at least not any less safe than it has been over the last several decades as programming crews has gotten much better.
Right. I guess when you describe it that way, software writing software is really just tools and tools are getting better and better.
And like you say, there's other aspects of programming that really need the nuance and the artistry of humans.
Now I have to ask, deeplearning.ai is your new startup.
You left Baidu a couple of months ago. Tell us all about it, or what can you tell us about it?
DeepLearning.ai is a project I'm really excited about.
It relates to deep learning. It also relates to AI.
And I hope to be able to say more about that in several weeks. well I had to ask and I know we're all fascinated to know more Andrew Ng thank you so much for joining us on the podcast thank you Michael
