Hello, and welcome to the Nvidia AI Podcast.
I'm your host, Noah Kravitz. Our guest today is the author of a forthcoming book that traces the history of AI from the mid-20th century to the present.
What makes this book like other great history books, but unlike some others about AI and tech in general, is that it's about the people and the stories behind the technology as much as the tech itself.
And it's really funny. Kate Metz is a reporter covering emerging technologies for The New York Times and his book, Genius Makers, The Mavericks Who Brought AI to Google, Facebook, and the World. is due out March 16th.
It's a great read, full stop. It's got great characters and as you might guess, some pretty jaw-dropping moments.
We're fortunate enough to have Cade here to talk about the book, so let's get right to it.
Cade Metz, welcome, and thanks for joining the NVIDIA AI Podcast.
Glad to be here. Thanks for having me. And I enjoyed your description.
If there's one thing I want to do with this book, I want to show people that It is about people in particular and that it is funny and that people, including engineers and AI researchers, are fascinating, unexpected in some ways, and funny people.
My father was an engineer and He used to joke about all the cliches that were applied to engineers. part of my mission with the book and part of my mission on earth is to show people that engineers Coders like my father are real, interesting, fascinating people, too.
Well, you mentioned when we were emailing, when we were getting this set up and you sent over a preview copy of the book, that the prologue kind of drops you right into the narrative.
And it does. And one of the things that I really loved about it was that it wasn't one of those prologues that sets you up with the story.
And then it like. changes tune when you get into the book proper and and it's no longer about the people it was just from the get-go it's kind of as much the story of this world-changing technology as it is about this man who can't sit down.
He has to travel cross-country without sitting down because of this back injury.
And then you tell the story of AI through these people and And all the skeptics and all the doubt they face and all this stuff.
So why don't we kind of, you kind of started with this, but why don't you give us the elevator pitch?
What's Genius Makers about? Well, the other thing I'm seeking to do with this book is show people the reality of what has happened with so-called AI what is happening and maybe what will happen, or at least point to it.
That term artificial intelligence gets thrown around so much. and it gets applied almost to anything and everything.
And that confuses people, I feel like. And it certainly confuses people You know, like my mother say, who's not steeped in this area, and it's hard. for people to understand if they just read what goes on in the press, what is really happening.
So much of it is hype. So much of it is nonsense.
So much of it is misleading. And you've got some people on one side saying, we're on our way to a machine that can do anything the human brain can do.
And then you've got other people whose qualifications are just as good, who are just as intelligent and just as well-educated and just as steeped in this area.
And they'll tell you We're nowhere close to that.
That's all nonsense. All this stuff about deep learning is way, way overhyped.
All the technological progress is about to end.
And if you hear all that, even if you're in the field, what are you supposed to do?
Right? Yeah. So my aim and, you know, I got halfway through the book and I thought, wow, this was a mistake.
My aim, though, was to try to encapsulate encapsulate everything that has happened over the past 10 years and explain it to people.
And, you know, miracle of miracles. By the end of it, I felt like I had gotten there, mostly because I found these incredibly interesting people who some readers may have heard of, they may not have. but I don't think they know the full extent of this story and who these people are.
And what I kept telling myself at my lowest moments was, If I can show people who Jeff Hinton is, and that's who you alluded to when you talked about the prologue.
If I can show people who he is, this book can work, right?
It's not about me or what I am doing. It is about him.
He is fundamentally and in surprising ways, time and again surprising, a fascinating person.
Let's get into that a little bit. And, you know, without...
I mean, spoiler alert, AI is still here, right?
So we're not going to spoil the story of AI, but without spoiling the details of the book, Who's Jeff Hinton?
Jeff Hinton was born in London just after the Second World War.
And He came up in the field of AI, and that alone is a great story.
But he came up in the field of AI at a time when it was at its lowest ebb, the early 70s.
Yeah. And the book goes into that, but he had this belief that you could build intelligent machines, so to speak. in the image of the human brain.
He had a particular belief about this and he spent his career, 50 years now, On that notion.
And over those 50 years, that idea rose in people's estimation in the field and then sharply fell off and then rose again.
He consistently believed in and worked on this one idea called a neural network.
You know, it's a mathematical system built in the image loosely in the image of the brain.
And for decades, literally, a lot of people thought this would never work.
And just at the moment where That idea reached its lowest ebb.
It started to work. And that's really... where you can trace the gains of the last 10 years to.
And that's what the book tries to document, and show what that idea is, why it struggled for so long, and then why it suddenly came to the fore.
And it does that largely through Jeff. And so there are these stories about, you know, going to these academic conferences and people shouting each other down and making fun of each other in front of their colleagues and And these characters, Marvin Minsky and Frank Rosenblatt, and these early things, with something called the perceptron and symbolic AI
Maybe can you talk a little bit about kind of those early days and kind of what, I don't know, the stories of...
Jeff's resilience and kind of his faith and his own belief and then these moments within academia at these conferences.
Can you just talk a little bit about that and kind of the things that had to happen just to get us to the point where the past 10 years could even have a chance of happening?
Yeah, I mean, what people may or may not realize is that the neural network idea that Jeff really latched on to in the early 70s goes back even further.
It goes back to the fifties or even the forties, you know, in some ways, but Frank Rosenblatt is, who was a professor at Cornell and then at a lab, a sister lab of Cornell in Buffalo.
He's one of the first people to build one of these things.
And he had he had incredibly high hopes for it.
And you could see this in the pages of the New York Times in the late 50s, where he He sort of lays out this extravagant and unbelievable path for a neural network. not only solving computer vision, but speech recognition and becoming able to to not only think, but recreate itself and fly into space.
It's astounding. Yeah. People tend to believe these sorts of things when an expert is talking about them, and there was a lot of hype around this idea.
And then the hype was debunked, and Marvin Minsky was one of the people who really turned against this idea.
They are a fascinating pair of researchers.
They have a history together. They went to the same high school.
Right. and end up on the opposite, on either side of this idea.
And they are a really good way of showing the tension here and showing the tension that has existed in AI since the beginning.
People think about it as a monolithic field sometimes.
The New York Times in the past has said, AI experts say, well, that's not one solid, always you know, always intent on agreeing with everyone else group.
That's a group that really disagrees. And you have these tribes within that and they often disagree.
Let me tell you, those two guys, Rosenblatt and Minsky, really show how that can happen.
I think especially for this podcast in particular, a lot of times we have folks on who... have just done something, whether it's a DIYer who, you know, did something cool at home that got some attention. or a startup or even an NVIDIA researcher talking about the latest and greatest.
But I think what's so great about your book is this kind of earlier history that maybe people don't know about or maybe sort of read about, but you forget because things are changing so fast these days.
But I want to ask you a little bit, just kind of latching on to what you were just talking about, about the AI winter.
And maybe you can describe briefly for the listeners. what that was, when it was, and how we got out of it.
Yeah. The AI winter is a common notion in the field.
And basically, the idea is These are difficult technologies to build, and you need a lot of money, and you need the backing of some really deep-pocketed organizations.
In the beginning, that was the Department of Defense and other parts of the government.
And they funded people like Minsky and Rosenblatt.
The Navy, among others, funded Rosenblatt.
And when Rosenblatt is making these grandiose claims and people believe it, the money comes in.
And then years go by, and people start to realize, wait, wait.
This is not working as it would work. Maybe we shouldn't fund this.
And then you got people like Minsky saying, This is definitely not working like he said it would, and it's never going to work.
And so then the money starts to flow to people like Minsky and then Minsky stuff doesn't quite work as, as he said it would and others.
And, and, And then the money goes away.
And that's when you enter these AI winters.
And then you have to wait a while until the technology progresses, but also until people forget that these overinflated claims can happen, and then they happen again.
And people believe them again. And you see these cycles throughout the past 50 years.
In a lot of ways, we're in the middle of another hype cycle, right?
A lot of stuff gets attributed to these technologies that is not really the reality.
But this is just how Silicon Valley works, right?
It's these hype cycles. It's been like that since the 50s.
There are a lot of stories and moments in the book where you kind of, you know, and I think it's this way with so many walks of life. read the history or, you know, listen to the people who are there tell the stories. these almost chance moments where it's like, oh, you know, if these two stars didn't align, maybe none of this would have happened. the Microsoft guy who says, oh, you know, if they'd hired me a year earlier, I might have accidentally shut the whole field down and all that kind of stuff.
There's this, um, This meeting in a Google kitchen between Andrew Ng, who, you know, is a name lots of people know, and Jeff Dean, another luminary in the field who You know, it might have been a chance meeting.
It might have been actually Andrew was arranging it the whole time.
And that's what kind of set off this next wave of harnessing you know, all of Google's data center computers to power what became the infamous cat paper.
What are a couple of your own favorite stories from researching the book, whether they made it into the book or maybe not? that stuck out to you?
Well, first of all, it's funny. Someone asked me the other day, what are the stories that you didn't put in?
They are all in. I'm so glad because there were a few that I was worried weren't going to make it because you really have to make sure they're right.
But like, first of all, the best story is the prologue, right?
Which is the moment that Jeff Hinton and his students move into Google.
And it's a, It's a moment around this technology that is now called AlexNet. after one of Hinton's students that has been written about ad nauseum in the tech press, sometimes in the mainstream press,
AI researchers know about AlexNet, which was extremely important paper that really launched neural networks into that field and into the mainstream a lot of ways.
It So people know about that, but they don't know the real story of how those three people, Jeff and his two students, Alex Krzyzewski and Ilya Sutskever, moved into Google.
And that's ultimately what this is about.
It's about people like Hinton who had this idea for decades being snapped up into these companies when their idea started to work.
And that is the moment when this happened.
And so that's my favorite story. And I spent years reporting that and getting the details right.
And I still almost don't believe it because it's such an amazing story.
All the players are there from the beginning, right?
Microsoft is there. Google is there. DeepMind is there.
A lab that had only just gotten started in London, which would rise to the fore in the other ways in years to come.
They're there and the conflicts are there.
China is there and their involvement. I mean, Baidu, the Google of China is involved in this.
That, to me, is an astounding story. And it's going down at Harrah's on the Vegas Strip, which anybody who's ever gone to a tech event or conference out there It just adds this wonderful layer of atmosphere as you're reading.
Yeah. So that's, that's definitely my favorite, but there are so many others involving deep mind, for instance.
And, uh, and how they moved into Google.
That is also an astounding story and how Jeff Hinton pops up in this.
The guy, he's everywhere, and every time he pops up, it's in these like Ways that you just cannot believe.
And there's so many little, you know, we mentioned kind of at the beginning about how the book is wonderfully funny and to kind of. tangent for a second because I didn't want to interrupt you before, but One of the things that I love about the book and I think makes for great reading on whatever the topic is, but particularly if it's a historical type of. manuscript or text is that you're able to write, you know, the book is written in a way that is, um, very readable, right?
It moves at a nice pace. And, and as we've been talking about it, it's so human and full of human stories, but you're packing in a amount of information and history about a wildly complex subject matter into these pages.
And as I was reading it, I was just thinking like, man, how did you distill down what must have been terabytes of data, so to speak? into a 330-page book.
Can you peel back the curtain a little bit and tell us about your process of when you got the idea for the book and how you started reporting it.
And if that bleeds into your work as a covering emerging tech uh you know for the times um all the better but but how did this all kind of get started and how did you go about reporting it well You know, I had started covering this area when I was with Wired Magazine and had covered it for about three or four years, the rise of neural networks. and got a glimpse of what was going on and a glimpse inside the lives of some of these people, including Hinton, and pitched a book along these lines.
And it was when the hype was really at its height and the book was bought.
And I think it was a good pitch. And You know, it landed the book a home, but the book as it progressed is nothing like the pitch.
Like it was that it was going to be about these people and it was going to be a narrative.
But what ended up happening is the story went in places that no one expected, including me.
And what it ended up being about was, you know, people like Hinton and their belief in this idea and their idealism.
Yeah. and how that clashed in so many ways with these very large companies which are driven by other motives This happened time and again in ways that surprised a lot of people.
And that really became the story. And That certainly surprised me.
I mean, just as far as putting it all together, You know, I couldn't have done this if I wasn't a beat reporter in this field and hadn't done it for years and really saw what was going on.
It was kind of a product of my job. But in the end, I think it got to a good place.
Have you met Jeff? Oh, absolutely. I talked to him today on the phone.
And were a lot of these—and, you know, again, without asking you to reveal sources, details, etc.
But were a lot of these stories related first person or third person or—
You know, how are the stories told to you?
Because the detail throughout is just it's very detailed.
Right. So just curious kind of how you went about reporting that stuff.
Well, that is just about going over the same ground over and over and over again. going back to people, trying to get them to reveal a little bit more, going to new people with that little bit of information you got from the person before and say, can you verify this?
What else can you tell me? Like, for instance, that lead anecdote, that prologue, like I said, that took years. to report.
And to me, one of the most difficult parts of it and one of the most important parts of it was the sale price.
That sort of shows you what's going on there.
And I was worried at one point that I wasn't going to be able to get the sale price into the book.
You've got to have multiple sources verifying that.
And this is just for the listener, this is the beginning when Hinton goes and eventually sells his company that gets him and his team into Google.
Yes. And, you know, I had, you know, I had the whole Jeff Hinton, you know, he sets up this auction for his company and it's just...
I had how the whole auction played out, but I needed to verify the price.
It just takes time. lot of doing especially because a lot of these people have moved into these very very large companies and And that makes them hard to talk to.
So it's just a lot about graft, as they say in the UK.
We're talking to Cade Metz. Cade is a reporter covering emerging technologies for the New York Times out of their San Francisco bureau.
But we're talking about his forthcoming book, Genius Makers, The Mavericks Who Brought AI to Google, Facebook, and the World.
It's due out March 16th, which is in about a month as we record this.
And again, just to say it again, A great read, highly recommended if you're listening to this pod and you're interested in any aspect of AI, let alone the history of how this stuff all come to be where it's at now.
But Cade, let's kind of fast forward a little bit closer to the current day.
And I wanted to ask you a couple questions about things kind of near the end of the book.
One is about robots. Can you tell us a little bit about how Peter Abbeel change Jeff's mind about reinforcement learning and what that might mean and what Peter's work means for the future of robotics?
As we discussed, Geoff Hinton believed in this neural network idea, this mathematical system that can recognize patterns. data.
And that's what drives image recognition today and speech recognition.
So when you're phone can recognize the commands you you speak to siri yeah and you know it is driving a lot of like natural language stuff.
But one of the things this has been applied to is robotics, including self-driving cars and their efforts to recognize, say, pedestrians on the road or that kind of thing.
And quick aside, people probably read the book, you'll be surprised at how early folks were working on self-driving cars.
Absolutely. And it's so interesting how all this stuff dovetails.
And it sort of happens at the same time people start believing in the self-driving car idea.
But basically, these systems that can learn tasks on their own. can be applied not only to self-driving cars, but they can be applied to other types of robotics.
And as you mentioned, What you can do is you can apply this idea to what's called reinforcement learning.
And so that's when a system learns literally by trial and error, extreme trial and error.
So like there's this great scene in the book where this other researcher who was a colleague of Peter Beals at the University of California, Berkeley, named Sergey Levitsky. folks might have heard the name.
Absolutely. So he goes to Google and, you know, they set up what they, what they called the arm farm.
And it's these robotic arms that, And they learn in this way.
So they just like put a bunch of stuff into a bin.
And the idea is to get the arm to learn to pick up the stuff, but it has to do it by trial and error.
So what it means is it just fails. over and over and over again.
And they go home for the weekend and they come back and it looks like the lab is covered in blood.
Right. And it's just like a crime scene.
And it turns out like it had, you know, tried to pick up and failed to pick up a thing, a lipstick.
And so there's just lipstick all over the place.
But this goes on literally for days and weeks. as these machines learn these tasks.
And even Jeff Hinton has been really skeptical of this reinforcement learning idea.
There's a lot of tension in the book between him and the people who believe in this idea, most notably the researchers at DeepMind, who we mentioned before.
But even Jeff Hinton comes around to this idea because he realizes that The amount of processing power needed to drive this, which is extensive, is now available in some ways and it continues to grow.
And that's what that's about. Like you just need enormous amounts of computing power to do that.
So even he, you know, sees some future in that idea, as extreme as it is.
So one of the things that kind of stuck out towards the end of the book is that Hinton talking about how old ideas are new.
And there's this idea that, uh, Well, there's kind of two threads to that one is sort of how that plays out throughout the book.
And then the other has to do with something that, if I have it right, that Jeff kind of first started playing around with this idea of capsule networks back in the 70s.
And now 40 or so years later, he's working on capsule networks again.
Can you talk a little bit about what a capsule network is and then also maybe just around this idea of old ideas becoming new again and how that's a thread through Jeff's work?
I'm glad you... you focused on this, you know, there's another great character in the book, a student, another student of Jeff's and students of Jeff litter the book, but a kind of George doll character.
One of my favorite interviews, he's now at Google, studied with Jeff at the University of Toronto.
And that's what he said is that, The theme in Jeff's lab at the University of Toronto before Jeff joined Google was old ideas are new.
And what that meant was is that if an idea had not been disproven, if there was still hope that it would work, if there was still a small chance that it would work and you believed in it, You kept working on it until you could prove it didn't work.
And that alone is a fascinating idea. And it's an idea that's at the heart of so many good ideas. novels, so many good nonfiction narratives, right?
People... who believe in something in spite of everyone else saying they shouldn't believe in it.
And that was the case here. And Jeff has continued along those same lines There's been a backlash in recent years, even in the AI community, against neural networks and people who really take a lot of pride almost in pointing out their limitations and their limitations are extensive.
But Jeff has this belief in these kinds of systems that certainly continues and is really part of who he is.
And one of these ideas is a capsule network. which is almost like an enhanced neural network that seeks to be more powerful than what has come before.
And it is an idea that he first had in the late 70s and he started to revisit it.
And there's a great scene in the book where he kind of demonstrates this idea to me.
Jury's still out on what the effect of this idea will be, but it's just one of many that Jeff, 50 years later, continues to work on.
And so to kind of land on the big question hanging out there for the initiated and not, will the robots take over?
There's always been this tension and it's in the book and it's in the mainstream media and in the informed and uninformed discussions about this stuff. between what people call practical and general AI.
There's this little anecdote, this little line towards the end of the book, about how, and talking about, is general AI, not even is it possible, but is it worth focusing on is that we're thinking about, or should we be thinking about more practical task-oriented things?
There's a line about, does a robotic surgeon really need to know the baseball scores?
Like who cares? I just want it to be good at surgery.
And so at the end of the book, you kind of note that Hinton maybe doesn't even really think that general AI makes sense.
Can you kind of talk about that a little bit and maybe throw your own perspective in covering emerging tech for so long? working on the book for years now.
And what do you think about it? Here's my main aim there.
And this is how I want to help people. when it comes to thinking about this.
And you're right, there's a lot of uninformed discussion and this can be a confusing topic, but what I want people to understand is that if someone tells you that they are going to build AGI, artificial general intelligence, a machine that can do anything the brain can do.
If they tell you that today, they don't know how to get there.
So on some level, that is aspirational, and people will argue about that.
But really, what this argument is about.
At this point, it's about a belief. And there are some people who are driven by this belief that this is not only possible, but possible in the near term.
And there are some people who say, wait, hold on.
How can you say that? Right. We don't have any evidence that we can get there.
And the chapter that tackles this is called Religions.
And some people see it that way, and I want to make it clear to people that It's almost like any other Silicon Valley company that comes to you and says, I'm going to build X. They're not there yet.
And they may or may not get there. Even with self-driving cars, it's not clear when we're going to get there despite the promises.
So when it comes to a machine that can do anything the brain can do, That's a task far, far more difficult than a self-driving car.
And that's what I want people to understand.
None of us know what the future holds there.
So, you know, any of us can say what's going to happen and who can say that we're wrong because none of us know.
So aside from the book, what are you working on these days?
Is there an emerging technology beyond AI that you're keeping your eye on?
Well, what's so interesting to me is that all the stuff that is discussed in the book, all these threads the book follows over the past 10 years, they continue to play out.
There have been some big moments since I finished the book DeepMind essentially solving what is called the protein folding problem, which is a key problem in drug discovery and other types of medicine and biology OpenAI releasing what's called GPT-3, which is this natural language system that does things that are unexpected,
Timnit Gebru at Google saying she was fired after she called attention to ethical questions in a lot of these systems.
All that happened since I finished the book, all of them are just a continuation of what happened in the book.
And in some ways they really echo these moments in the book.
And what it shows is that these technologies are continuing to progress. at a incredibly fast rate.
And the questions that they raise have not been solved.
And we as a society, Google as a company, OpenAI as a company, so many other companies are still struggling with all that.
And we're all trying to determine amidst this larger you know, sort of sea of questions we're all dealing with as a society right now, how we're going to deal with this.
That's what I cover. That's what's so interesting to me are these ideas, not only in and of themselves, which are fascinating, But what are their effects on the larger society?
And that's what I cover on a daily basis.
Again, for anybody listening who has not read the book, it's well worth the time.
It's a great read. It's entertaining. It's fun.
It's human. And you learn a lot about the history, but also just to echo how you just put it, kids.
The human element to all of this and what it's meant and what it means going forward.
It's a great story for folks who want to follow your current work. with the New York Times and elsewhere.
Do you keep a blog? Are you on social media?
Where can they kind of latch onto your train and see where it's going.
Uh, main place to follow me is, at the New York times.
Um, but I'm also on Twitter at Cade Metz.
Thanks for having me. I truly enjoyed it.
Likewise. The pleasure was definitely ours.
And, you know, thank you again. Hope everybody out there gets a chance to read the book.
I feel like I should stop shilling, but it's really just an enjoyable experience.
So, Cade, thanks for taking the time to come on the show and all the best in your future work.
You too. Thank you. Thank you. Thank you.
Thank you.
