Welcome to Nvidia's AI podcast. The robots that have taken on tasks in the real world which is to say the world where physics applies are primarily programmed to do a specific job.
Weld a joint in a car or sweep up the cat hair.
Programming a robot is distinct from robots learning to do something.
So what if robots could learn? And take it a step further.
What if they could teach themselves and pass on their knowledge to other robots?
Where could that take machines and the notion of machine intelligence?
And how fast do we get there? Those are the questions that our guest, Sergey Levine, an assistant professor in the Department of Electrical Engineering and Computer Science at UC Berkeley. is working on finding the answers to.
Sergey, welcome. Thank you. So, Sergey, Do I have that right?
And we know about deep learning, and we know about it in the sense of teaching systems to recognize images.
But in the physical world, you're trying to teach robots to do certain things.
And why is that a different problem? And how did you get to it?
That's a very good question. So I guess let me start with why it's a different problem.
And then I can talk about how I got to it.
So Usually when we build deep learning systems to, let's say, label objects and images, The way that we do it is we first ask people, usually quite a large number of people to look at a large number of pictures and tell us what's in that picture.
Oftentimes, this is done using web services like Amazon Mechanical Truck.
The trouble is that if you want to use an approach like that to control a robot, you can't just show pictures to people on the web and ask them to write down the torques that the robot needs to apply to its arm in order to perform a task.
That's just not something that people can tell you.
So if you want to get a robot to do interesting things, You kind of needed to learn on its own because this is not something where people can just give you these perfect labels like they do for image recognition.
And so robots, in that sense, learn differently than we do, it's fair to say.
Well, in some sense, they actually learn more like we do.
They learn differently from how our image recognition systems learn.
We learn by trial and error. We learn by actually trying things often failing at those things and getting better and better.
Whereas our image recognition systems learn by being kind of slavishly supervised every step of the way, being told exactly what their answer should be and using that as training data.
So the robot problem, and I love this example of you're torquing a screw or a nut or something.
Again, we learn by trial and error, or we use the right tool, I guess, and says, okay, when it hits this limit, stop.
But to teach a robot to do that, what do you have to do?
There are a couple of things that you need to do in order to teach a robot or enable a robot to learn something on its own.
One of the most important things is that you have to somehow communicate to the robot what it means to succeed.
That's one of the most basic things. Perhaps you don't have to tell it exactly how to do something, but you need to tell it what it should be doing.
In the parlance of reinforcement learning, this is called a reward function.
So a reward function... is a function of the state of the system that tells you how desirable that state is.
So if the robot is trying to assemble a car, a highly desirable state is one where the car has been assembled.
If the robot is trying to drive an autonomous car, then a highly desirable state is one where the passenger safely arrived at the destination.
And I know reward for us humans means, okay, I get a bonus or a paycheck or a cookie.
What possible reward can you give to a robot?
So the reward is a piece of code. So it's a piece of code that gives the robot a scalar valued function that tells us how desirable status so sometimes it's called utility.
Maybe what your question is getting is, where do you get this function?
One way to do it is to program it by hand, and some tasks make this easy.
For example, if you have an autonomous car that needs to reach some destination, then maybe you say, okay, if the GPS reading is within some threshold of a desired GPS reading, then you've succeeded.
But in some cases, it's actually very difficult to program rewards by hand.
So for example, if you want to have a robot learn how to pour a glass of water for you, Just measuring how much water is in that glass, like to see whether it did the job right, that's hard.
So one of the things we've actually explored is reinforcement learning. where the reward function itself is also actually learned.
So you observe people doing a task, try to infer what are the goals that these people are optimizing, and then the robot can attempt to optimize that same goal itself.
I see, so let's say we are trying to teach a robot to fill a glass with water, You observe people filling those glasses and doing it the right way.
And then the robot sort of reverse engineers what success is.
That's right. So the robot looks at all those demonstrations and then tries to figure out what is it that these demonstrations have in common, what is it that the people in those demonstrations are trying to achieve.
And demonstrations can be provided in a number of different ways.
The holy grail of this field of research is to be able to get demonstrations that are just unstructured raw videos of a person doing something.
That's difficult, but that's sort of where this research is going.
More conservative versions of this technology do something like They take in a demonstration provided through kinesthetic teaching where a person actually moves the robot's arm.
So you can imagine like if you're learning a golf swing, maybe someone will actually like guide your arms through the motion.
So it's kind of like that. Or through teleoperation, where a person sort of remote controls the robot, and that gives it a demonstration.
Clarify for us, because you mentioned a golf swing, right?
So we know that there are these kind of robotic golf swingers. where they're testing golf balls or they're testing something, right?
And it just swings it the exact same way over and over and over.
But how is that different than... a robot actually swinging a golf club with the intention to play golf in some sense?
If you imagine Building a robot to play golf on a very particular golf course with a very particular position of the ball, that's actually not all that hard.
You just have to have that robot memorize a motion and just repeat that motion over and over again.
But if you want to have a robot that can play golf on any golf course in the world using any golf club that it has to pick up and hold in its own hand, that's incredibly difficult.
This is what's referred to as generalization.
It has to be able to use what it saw in the past, in particular instances of a task, to solve new instances that are different in a visual and physical way.
Generalization has been studied in great detail in areas like computer vision.
Computer vision is all about generalization.
When we build a vision system to recognize objects and images, If we use, let's say, pictures of a particular house as training data, it's not that we want to recognize that particular house.
We want to recognize all houses, so we want generalization.
It's the same way with robotics. If we want robots to be effective in open world, realistic environments, we want them to generalize.
We don't want them to just do the same thing that they did during training exactly.
We want them to understand what about the task really requires certain movements or certain adaptation.
How did you personally get to this? Why is it that you want to help robots to learn?
Is it Were you fascinated by robots? Did you want to have a robot companion or just feel like...
Hey, they could do a lot more. How did you come to it?
It's actually kind of a funny answer. So when I started graduate school, I actually wanted to do computer graphics.
I was a big fan of video games, movies, and I thought that graphics was really fascinating because it was a vehicle for simulating the real world.
And one of the most interesting parts of the real world to simulate is the humans and animals that inhabit it.
And if you want to simulate humans and animals, then you have to simulate their brains.
And that's basically the artificial intelligence problem.
So as I was working on graphics, computer animation, one of the things that I realized is actually the most interesting aspect of this problem is the artificial intelligence that goes into it.
So then I decided that I would switch my work over to focus entirely on artificial intelligence. developed some methods that were applicable to character animation and computer graphics, and then decided that I would try my hand at applying similar techniques to control real-world robots because that seemed like a very natural kind of next step for that technology.
Second, you talked about being a gamer, and you talked about your initial foray into this world came through computer graphics.
But did you have an encounter with a robot in fiction or as a child?
And is that what kind of inspired this direction?
There's this children's movie that has a robot called Baymax.
I don't know if you've heard of this. It's a robot. that was built actually by a graduate student at a university for the purpose of helping patients for medical help.
The thing that I really liked about that movie is that it's an action movie.
The robot goes out and fights the bad guy and so on.
But this idea that scientists can build a robot as an experiment, but for the purpose of actually helping people, not for the purpose of fighting the bad guy or for the purpose of breaking things, but specifically for the purpose of helping people who need, in that case, medical assistance.
The fact that something like that makes it a popular culture and a lot of people see it, that to me is actually quite inspiring.
I think we should be thinking more about robots in this way.
Yeah, popular culture tends not to, at least definitely not here in the United States.
Robots take on much more of a... battle bot kind of quality um yeah and i think that uh you know we should I mean popular culture does affect how we think about technology and we should really think about how technology can help us lead better lives because it can and if all we think about is robots fighting wars and breaking things, then maybe it'll be harder for us to find the imagination to really imagine how they can be helpful to us.
Was there a moment or an incident or an event that made you think, ah, we could take what's been done or being done? you know, researched in the digital and kind of image recognition realm or voice recognition or whatever. then apply again to to the real world to a place where physics apply around 2000 when I, when I was a graduate student, I actually attended a lecture by, uh, Andrew Ng.
What Andrew Ng talked about in his lecture was, um, this idea that you can have a class of algorithms, a single class of learning algorithms that can solve all possible learning problems Uh, and he gave some examples of deep learning algorithms that could recognize things in images that could recognize sounds and audio. and even could model haptics touch.
And he was talking primarily about perception, but I was sitting there listening to this lecture and thinking about the latest character animation problems that I was dealing with where the main challenge was actually how does this virtual character react to the environment, react to the physical events?
And I was thinking, well, if there is a single algorithm that can do all these perception tasks, is it really just for perception?
Perhaps this class of methods can be extended to the point where it does perception and action altogether.
So that lecture was really the thing that inspired me to, I think, to study these techniques.
You know, this is a question that we – and we had Andrew Ng on the podcast.
We talked to a lot of people in this field, and I just wonder, like, you – speak about this and think about it differently than Andrew.
You certainly have overlap. But what would you say that you're good at?
And how has your mind and your creativity kind of pushed where you're going in the field of robotics in different directions than They've gone before.
I tend to think about problems in terms of the... kind of the end goal that the entire system is trying to accomplish.
When you think about a computer vision system, the job of a computer vision system is not really to output class labels.
It's that it's going to output something and some downstream system is going to act on that and some system after that.
Cats in and of themselves aren't the goal, is what you're saying.
That's right, yeah. I don't think I'm by any means like the... the main person who does this or the first person to do it or anything like that.
But to me, this is like really important because I'm, I like to think about how the whole system is going to actually interact with the world and what the final result of that interaction will be. you know that way of looking at it I think is very natural for roboticists or for Actually, also for computer graphics researchers.
But oftentimes, that's not how machine learning people think about it because they often prefer to sort of pick a crisp problem and then say, okay, here are the inputs and outputs and we'll just figure out the stuff in between.
Of course, this is a bit of a caricature, but I think that's one of the differences between kind of the robotics approach and maybe a more traditional machine learning approach.
What is so hard about what you have to do and what are the hurdles in front of, you know, you and others in this area of research? having a robot do something like picking up a cup is incredibly hard.
And I think there's a bit of a, almost like a cognitive bias that we have where we don't quite realize how difficult it is because Our bodies and our minds are evolved to be incredibly good at basic physical skills.
That's why we call them basic, because they're basic to us.
But we're, in some sense, optimally adapted to do those things.
We're adapted to interact with our environment.
That's what we're good at. So it's sometimes a bit hard for us to understand just how difficult these things are, Not necessarily at the physical level, but at the cognitive, at the mental level.
The kind of... accuracy of perception, the fluidity of actuation, the accuracy of your feedback control that you need to do what we would consider basic physical tasks.
You talk about the end goal. Let's talk about the present.
What can be done today? And how quickly does this take us in a different direction, this approach?
So I think that today, the majority of robotics technology that is actually out there making an economic impact is the kind of technology that we've had actually for decades.
It's the kind where the robot repeatedly does the same thing over and over again.
However, right now, I think we're right there at the cusp where Learning is really making an impact on robotics in a way that Essentially, it's just beginning to really translate into the real world, into things that people might actually use either in an industrial setting or even a service setting.
I think we're seeing some of these advances with drones, with quadrotors.
These are consumer technology, many of which are powered by some form of AI.
And I think we'll begin to see more and more of it in robots that also actually interact with the environment, that have arms, perhaps initially to a limited extent in sort of industrial or service sectors where there is some amount of human oversight. where there's somebody to basically fix things if the robot makes a mistake, but gradually these systems will become more independent.
I think learning-based technology It's one of those things that kind of doesn't work until it works because it depends in a really big way on the amount of experience that you can collect and the diversity of that experience.
So the first learning-powered robots that we'll see out there might be pretty clumsy and might require some amount of human intervention to help them when they mess up.
But once those systems are out there, once they're gaining experience and getting more and more proficient, their proficiency at those tasks will increase and they will actually become better able to perform their jobs with minimal help from people,
And they'll just continue getting better, which will make them a lot more useful, make them usable in situations where maybe there isn't a person to help them out if they mess up.
Pardon the interruption, but we have a favor to ask.
Leave us a review on iTunes, Google Play Music or whatever your favorite podcast platform of choice is.
It helps more people find us. As always, thanks for listening.
Now, back to the good stuff. Give us some of those situations as you imagine it.
I'm just wondering, in that kind of vision of what tasks or what scenarios they take part in, let's say it's all working great, there's this sort of generalized learning robot, it shows up and it does what?
I think the kinds of things where we'll see robots making a big impact will be jobs that either is difficult, unpleasant, or dangerous to do for humans.
Some of these jobs are jobs that are just not being done now.
Some of those are jobs that are being done, but we'd rather not have people doing them because they're dangerous.
For example, it'll probably make it a lot more practical for the elderly and the disabled To have a robot help them around the house, to help them with their daily lives, with their tasks of daily living. for basic kind of chores around offices, hospitals, and homes. and also for tasks that are dangerous, things like disaster relief, maintenance, industrial maintenance tasks that are done by humans that involve going into situations that are perhaps risky for those humans, like maintenance of nuclear power plants, for instance.
Those are I think things where we'll see a big impact and the kind of impact that will actually make people's lives better.
And what stands in the way of that? Is it the algorithms?
Is it the data? It's probably not the mechanics, but maybe it is.
You know, Or is it just the process of teaching them to teach themselves?
I think it's largely not the mechanics. In fact...
Tele-operated robots can do some really fantastic things.
If you've ever watched a video of an expert surgeon controlling a surgical robot.
Right, right. It looks like science fiction.
It's absolutely stunningly amazing. So the hardware obviously can be better, but it's there.
That's not the bottleneck. The bottleneck really is our ability right now to develop the kind of algorithms that can really learn in the complex real-world environments these robots will encounter, And then it's going to be, once we've done that, then it's going to be the data, the experience.
Right now I think we're still at the stage where we're bottlenecked by developing the algorithms.
But I also think we're right at the cusp where in the near future, we'll see the technology progress far enough that those algorithms will be there.
How do we then as people integrate them into our lives and our work?
That's a very good question. That's something that we'll have to deal with.
I think there will be some settings where it's very natural.
I think there are, for example, settings that are a little bit more industrial that are similar enough to domains that have already been automated, where it'll be very natural to add automation.
For example, helping stock shelves in supermarkets or warehouse automation.
When it comes to robots that really are doing their job next to humans in settings like the home or the office or the hospital, Yeah, like there might be situations where people feel like they would rather that a person do that job, not because the person is sort of better at it in some way, but because that's just a job they'd rather have a person for.
On the other hand, I think There will be a need in society, in some societies more than others.
Certainly in the wealthy nations, the population is aging.
As people get older, there's going to be more need for essentially for service robots or for people to perform those jobs. there might not be enough people for that, or the people might be too expensive.
So at that point, the availability of those robots is going to provide benefit to people who otherwise wouldn't get the benefit of that kind of help.
What do you want a robot to do for you? And then, and it doesn't have to be, it could be anything, but like if you know, and I'm sure you have things working in the lab, but, um, What is your ideal robot companion doing for you?
I think that for me, the ideal robot would be one that is flexible enough to do things that I request.
So I don't think it's about any one thing.
I think if we want a robot to do any one thing, that's actually not all that hard.
The trouble is that we never want just one thing.
What we really want is we want to be able to come home and say, like, can you do my laundry now, but for this shirt that I prefer not to go in the washer.
I'm going to need to get it dry cleaned or something.
Please set it aside and please leave them on my office chair or something so I can find them in the morning.
So it's not the particulars of the task, it's the flexibility and the kind of the customization that you can actually get from a system that is intelligent that you cannot get from a system that just from an appliance essentially.
Yeah. And it's the complexity of our human kind of needs and desires to where, you know, abandon that halfway through.
What I really want right now is, you know, something else where.
If it's a welding robot, all it can do for you is weld you a car.
I think that's maybe part of the appeal of some of these systems that have been that have gotten popular recently, like Alexa.
These things, they're not even robots, they're just essentially conversational agents.
But the fact that they do allow for some degree of customization of your requests, I think that That's something that really appeals to people.
One thing that fascinates me is this idea of robots teaching other robots.
And I have this kind of image in my mind of We're all about sitting around with cups of coffee and chatting about what they learned the other day.
How is that possible? This is, I think, actually a really important idea and one that will make a huge impact in when learning based robotic technology actually gets out into robots that are in the real world. is that unlike humans and animals, for a robot, you can basically copy and paste their brain.
You can take what they've learned and you can just make an exact copy of it.
And you can also copy and paste their experience.
You can take their experience and exactly losslessly transfer it to another robot.
So what that means is that If you build a robot that goes into the home and helps people, let's say, do their laundry, and you have 100,000 of these robots or a million of these robots deployed... they're actually experiencing 100,000 to a million robot lifetimes all at once.
Because they can pool their data, they can transfer it over the internet, they can combine it.
And they can have the benefit of that breadth of experience to become better at the job that they're supposed to be doing.
So if it requires a person 10,000 hours to learn a complex skill at like sort of maximum expert level proficiency, Maybe it'll take 10,000 robots one hour to learn it.
This is part of why I said before that robotic learning is one of those technologies that sort of won't work until it does because the kind of results that we might get in a small-scale laboratory experiment They might prove out our technology, but in the end, the really big benefit, the really amazing generalization will come from when there are many of them actually deployed in the real world, learning from their shared experience.
Well, I have to say I can't wait to see it and experience it myself.
And also, you know. have it respond, like you say, in this flexible, customizable way that I think we as people are all so used to.
Your work is really fascinating and, um, Keep it up, and we will keep an eye on things and hope to talk more about this.
Sergey Levine, thank you so much. Thank you very much.
