Cars that drive themselves, and let's be honest, more importantly, drive us around, is where a great deal of the excitement and focus is in artificial intelligence. and it's the subject of this segment of NVIDIA's AI podcast.
To discuss this fast approaching world, we have with us from NVIDIA's automotive team, Danny Shapiro, Danny welcome.
Thanks. At this point in what we read and hear, it seems like it's a foregone conclusion that In the not too distant futures, our cars will be driving us around.
Let's break it down. How does AI, artificial intelligence, and deep learning relate to the task of driving and And how is it a problem that we're able to start to solve today?
Driving, as you know, is a very complicated task.
There's a lot of things that happen as...
We drive our cars and that we have to react to.
There's really no way that we could write a computer program to handle the near infinite number of things that happen.
And so this is where artificial intelligence is critical.
What we need to do is first train the car, much like you learn how to drive, we train the car how to understand what happens in the environment and what is the right behavior for the vehicle to safely navigate.
We have to take the car out on the road and collect data.
So you see cars all over Silicon Valley and other parts of the world now that have an array of sensors on them that are driving around and they're collecting information, cameras, radar, ultrasonic. as well as LiDAR, which is a laser based system for sensing the environment.
So these are the things I actually I was walking through downtown San Francisco the other day and I saw the Uber car and Uber car driving around with this kind of ugly.
I shouldn't say ugly, but to me, it seemed a little ugly, cowling on the top that had all sorts of stuff packed inside.
Google has, we've seen them driving around for years now with different things on the roof.
That's absolutely right. sensors are generating a massive amount of information and that is coming into the car.
So the first phase really is we have to train our system how to understand the environment.
And this is where deep learning comes in.
You know, cameras will take pictures at 30 frames a second.
And so it generates a massive amount of information.
We use that information then to train the car in advance how to recognize another car, how to recognize a pedestrian or a truck or a bicyclist, and be able to discern, really, what is something it needs to pay attention to and worry about, and other things it could ignore.
Maybe there's somebody standing on the sidewalk that's not even in the range of the vehicle.
So we will ignore that. Is it all images?
And how do the images then connect to actually the rules of the road? and getting from point A to point B?
So it's not just images. Different car makers are looking at different types of sensors on their vehicles.
There's really a variety of different techniques.
And the reality is having a variety of sensors is very good because just like you have eyes and ears, you're getting different types of information from those different senses.
Um, you're able to then combine those to have a more accurate picture.
So if we use a camera, it's very good at giving us information that we can visually discern the difference between an ambulance and a delivery truck. whereas a radar would not be able to tell us the difference.
The radar can give us very good depth and speed information, but we wouldn't know much about that view.
I see. So it knows that there's two objects.
They're moving at different rates, and it maybe even sees other objects that aren't moving at all. but it doesn't know what they are.
Yeah, so together with these different sensors, and with the AI brain inside the car, we're able to build a very accurate picture of what's happening around our car.
It's almost like a video game if you think about it.
Inside the brain of the self-driving car It's recreated a full three-dimensional environment much more accurately than you or I could possibly do it.
We're monitoring a full 360 degrees around the car, and we can track... any number of objects, and also now using deep learning, we can understand what those different objects are.
Is it a person? Is it a bike? Is it a car?
Is it a truck? What kind of car? What kind of truck?
And then we've also been able to teach the system what type of behavior these different objects exhibit.
And so we know the speeds at which pedestrians move versus cars and trucks.
And we essentially have a full understanding of everything going on around the car so that we can very accurately, very safely pilot our vehicle through. this environment.
You mentioned all these different sensors.
Tesla showed how it's got these eight camera array that has this 360 degree view.
And I get how there's lasers and LIDAR and other things.
That sounds to me like sort of full awareness, right?
I have 360 degree awareness. But that still doesn't mean that I can drive.
So how do you put together that awareness with the sort of act or utility of driving? taking me to work or taking me to dinner.
You're absolutely right. It's a very complex and a kind of a whole pipeline of processing that needs to take place inside the car. in just a fraction of a second.
And so it starts with taking that sensor data bringing it in and understanding the environment.
But we also need to have a map and what's often called an HD map, a highly detailed map.
So this goes beyond a basic Google map that says you're at point A and you take a left here and you go straight.
These maps have information about the exact width of the road, the number of lanes, which lanes are turn lanes or straight. where the stop signs are, where the stoplights are, a lot of information.
And so basically what we'll do is then we'll use the information that we're sensing We'll couple that with this HD map, so we'll have very precise location of where our car is located.
We know where we want to go, so we're able to do general route planning that way, but In real time then, we're taking additional data in that helps us understand the other obstacles, whether they're fixed obstacles, or moving obstacles on the road and then we'll calculate a path.
It's called path planning, essentially. And then we will adjust our speed. and adjust the steering to move through that environment, staying within the lanes, avoiding any potential hazards and then also obeying the traffic laws.
So if there's a stop sign, we recognize we have to come to a stop.
And then we're able to analyze the situation, what other cars are in the environment, what people may be crossing.
And we will then wait until it's safe to proceed and move through that intersection.
You mentioned a stop sign. How does even...
We all know from, I think, a very young age that stop signs, at least in the United States, are red and they're octagonal or they're of a certain shape.
If you can break it down, how does this learning system know what a stop sign is?
It's a great question. And what if there's a tree branch hanging in front of the stop sign?
So this is the beauty and the power of deep learning, of artificial intelligence.
What we have done in the past and maybe what a smart camera does, it's based on computer vision. was to have engineers write software to specifically decode images coming into the car to identify that's an octagon. or this is red, or it has these white letters on it.
So there's an incredible amount of code that had to be written specifically to identify that sign.
And then we would have to multiply that by every sign we needed to know.
And like you said, if a stop sign looks different in another country, you'd have to go write even more code to figure out what that stop sign was in another language.
Instead, using deep learning and what we've helped pioneer with our GPU is the ability to do this very fast.
Instead of writing code, the data creates the software.
And what I mean by that is we teach it by feeding it information.
We would give the system hundreds of pictures or thousands of pictures of stop signs shot from all different angles at all different times a day. in different weather conditions, sometimes maybe with people blocking them or trees blocking them so you wouldn't see the full stop sign.
And the deep neural network then that's created forms an understanding of what a stop sign is.
And again, it's really like a human child would learn something the first time you give them a tow truck.
They don't know what that is, and you teach them it's a tow truck.
They learn that vocabulary. The next time you'd give them maybe a fire truck, they might call it a tow truck.
And you say, no, that's a firetruck. And so they understand that these things are related, but they're also different, and they build that vocabulary.
So through this process of training through feeding information, the deep neural network is able to then understand and recognize what these signs look like.
And so they can process it in a fraction of a second.
So the video that's coming into the car now can be analyzed and we can identify the stop sign or a stoplight, or a pedestrian, or any collection of these things in the frame.
And so that learning happens, like you say, in the same way that we learn where there's a teacher... but I'm using air quotes that says, you know, actually that stop sign, you got that one wrong.
And it goes back and tries again and again and again.
It's like, So this is a process that takes place in the data center.
So the training does not happen in the car.
Right. So we'll go out and we'll shoot a lot of video of driving scenarios or you might just have images of these signs and we'll use that as the data input.
That training process could take days, weeks, months to fully train a vehicle.
But to show you the power, I want to cite an example.
One of our partners, Audi, had been spending two years developing a smart camera, so a camera that goes behind the mirror in the front of the windshield. and takes an information.
They'd spent two years developing software with a partner to recognize, Street signs, the exact example we were just talking about.
And so they had to write code to basically figure out what each of the signs could be.
Using deep learning in our system, they spent four hours feeding their massive database of German street signs into the system.
And after those four hours of training, no coding was involved, just loading data, they were able to achieve a better level of perception than two years of engineering development on a smart camera system.
I love this very painstaking, like, this is what a sign looks like, what you What you're supposed to do, what it means, I see.
And so the beauty of this system is now if they want to not train it on German street signs, but instead understand the vocabulary of Japanese street signs, The engineers don't have to go back and start writing code, but they have their massive database of images of all the Japanese street signs.
They feed that into the system, and now the car can understand how to interpret. signs in Japan.
Denny, I imagine that you've had the opportunity to ride, be a passenger in a Tell us what it's like.
Is it like driving with your wife or your husband or your brother or your sister?
It's an amazing experience. And once you are in that vehicle, you recognize how the systems in that car are so much safer than any human possibly can.
I don't doubt that for a second. There may be this momentary hesitancy or uncertainty, but as soon as the car starts driving, it is so liberating.
It is so incredible. And so I cannot wait for this technology to be available.
But let me just ask you why. Is it because there's micro-adjustments or because you...
You see this thing, this driver, this deep neural net is driving you in such a way that You're like, oh my gosh, they are so much better than I am.
Well, I mean, you don't see anything other than the steering wheel spinning on its own, which is kind of a fun novel thing to experience.
But you recognize that the car is able to perceive everything around you.
And it's able to drive with precision, um, But the tests that I've been in have ranged from those early days where it is a little herky-jerky now to the point where it's a very smooth, very comfortable ride.
And it just gives you that freedom to relax in the car very quickly.
I've actually been on some cars on a racetrack.
So... Oh wow. Sitting as a passenger driving over 100 miles an hour around tight corners is just a remarkable experience.
Seeing these cars. drive with nobody in them and then getting in as a passenger and nobody touching anything really instills an incredible amount of confidence in how good this technology is.
When you're in the car, how quickly for you did it go from this mind-blowing the steering wheel is moving itself to oh well the steering wheel is moving itself now i'm gonna check my email or read a book or take a nap.
Unfortunately, I wasn't in the car for more than a couple of laps when it was at the track.
The novelty is definitely there. It's an incredible experience, but I could foresee extremely quickly fully trusting these systems to the point where I would love to have it on my car for my daily commute so I don't have to deal with with traffic and and ultimately when cars will be able to drive themselves with nobody in it arriving at my destination and sending the car out to park and then being able to summon it back with my smartphone.
And again, I don't think we're too far away from that on the timeline.
We have traffic jam assist programs coming out with Audi next year.
We're working with Volvo on their drive me project, which is putting. their XC90s on the road in Sweden and UK and China for pilot projects with actual drivers.
And then we're going to see just an incredible rollout of the technology with other partners. in the very near future.
So much like you become dependent on your backup camera today or the blind spot monitor beeping, if you get in a car that does not have that, you miss it really quickly.
And I think it's going to be exactly the same.
Once somebody experiences some of these autonomous features in the car, whether it's a car that they've rented, a car that they have for an Uber, or a car they've ridden in as a friend because they just got a new Tesla. it's going to be hard for them to go back to not having these features because they really do transform the driving experience.
Is there sort of the platonic ideal of a driver out there?
It's certainly not based on me and people I know, but, you know, Is there, it'd be interesting to, is there a kind of the ultimate driver that's been kind of programmed and, or do you, get to adjust.
I guess if you're driving in Germany, you get to drive a lot faster than here.
I imagine that once... autonomous vehicles are fully in production, that there will be modes or different ways that they can drive.
Just like you have different suspension controls in your car.
You can have comfort mode. You could have sport or economy, right? and potentially different autonomous vehicles from different brands will drive differently.
You have some higher end, whether they're luxury brands versus economy brands, they may, again, have different ways of interacting.
But I think the key thing here is that Each automaker, their brand is going to be all about how their self-driving car performs.
How safe is it? Where can it drive? What automated features does it have in terms of is it just on the highway?
Is it in city streets and all the way up to, can the car drive around with nobody in it?
And again, I think that's what's happening today is you have virtually every automaker.
You have the suppliers to the automakers that are working on this.
We have startup companies here in the Valley and elsewhere, as well as very large computer companies and internet companies that are very active in this space.
I think they've all recognized this is a massive computing problem that they need to solve.
And they're all essentially using deep learning now to solve it.
So the flavors of drivers and kind of what the utility they might offer. or even the experience they offer might be different.
That's fascinating. I didn't think about that.
Let me just ask you this. You said driving a car with nobody in it.
Why would I want to do that? Well, imagine you wanted to, if you don't have a car and you wanted like a car sharing service, the car could just be delivered to you with nobody in it. and you get inside and maybe you're driving it or maybe you're riding as a passenger in it.
Maybe instead of flying to LA from San Francisco, you hop in the backseat of a car and it drives you there automatically.
Well, let's talk about that because what you're describing as a sort of fundamental change to car ownership potentially, but to how we move around in the world, for sure.
I mean, how do you think about how this might change Our roads, for sure, but even our cities and our neighborhoods.
It's going to be gradual, but it's happening very quickly.
We already have partially autonomous vehicles in development. in tests all over.
And again, the work we're doing with Tesla Motors, Using our technology, they're putting into production cars that will be capable of fully driving themselves.
So the timeline is now. I think that in the near term, that people will become introduced to this technology.
And as soon as they experience, they recognize how fantastic it is.
So from the standpoint of just even having a traffic jam assist on Highway 101 here in the valley.
If you can enable the car to maintain a safe distance in front of it, go and stop and go traffic, stay in the lane, that's going to be a huge stress reliever for the occupants of those vehicles.
And then when they need to exit the freeway, they take over control.
So that's kind of this first phase. We're seeing some of these traffic jam assists.
And then we're going to proceed into kind of full chauffeur modes were point to point types of driving and eventually full autonomy.
Throughout this process, we're going to see a dramatic reduction in the number of accidents, number of injuries, and fatalities.
And to us, that's a huge motivation for doing this.
We're going to make our roadways much safer.
It's going to reduce congestion on our roadways.
Traffic will flow much more efficiently.
So there's huge productivity gains. Is that because we as humans just are... bad, inefficient drivers?
Absolutely. We're very distracted. Right.
More and more, as you drive down the freeway, you see everyone on their cell phones.
The accident rate is actually going up now the last couple of years.
It had been declining for decades and now it's on the upswing.
There's just an incredible amount of driver distraction these days.
So we're going to eliminate the human error portion, which accounts for over 90% of accidents.
And so again, there'll be great efficiencies on the roadways, reduction of accidents.
So I think we'll see a boost there. And then productivity gains.
Now, if people can do other things while they're on the road safely.
Which they do now, sadly. Yeah, but now safely.
I think the key thing is if you want to read, if you want to... watch a movie, if you want to get work done, check your email, whatever it is, you can safely do that while you're in the car.
So this final phase where we get to full autonomous driving, where a car comes, picks me up, or I just get in my car and it goes.
What are the hurdles there? And I've heard...
There's got to be a billion road miles driven to get this, you know, for us to learn enough to kind of account for every eventuality.
And I know you get this question all the time, and it's a squishy one, but how soon can I stop driving is really what I want to get to.
And what are the really tough things ahead of us?
You're absolutely right. We've done some of our own tests in our R&D.
And just for sake of developing some of our algorithms and training our neural nets, we did 3,000 miles of driving.
And we're not advocating that's all you need, but we were able to achieve incredibly good results after 3,000 miles.
And we're using that again to prove our technology and to showcase what it's capable of doing.
As we work with automakers, clearly they're going to train on hundreds of thousands, if not millions of miles and beyond. train the system, however, it's not just about the miles, but it's about the miles of unique So we just train it on the highway.
Fine, we can drive on the highway really well and there's less obstacles or less things to worry about.
But when we take the car into the city, it's not going to be able to perform like it did on the highway.
So we need to ensure that we are accounting for the jaywalker.
The bicyclist that swerves in front of us, the child playing in the ball rolling into the street scenario, all of these things.
And driving around, you rarely encounter those.
So it's not so much even about all those billions of miles. that they're projecting you need to capture those kinds of scenarios.
But instead, what we can do is focus on training our system using simulation.
And what's interesting is this kind of can get back to Nvidia's heritage in video games.
We can use... computer graphics and the kinds of technology that's used to create video games now to actually train neural nets how to avoid accidents and dangerous situations.
So we can simulate Children running in front of the car without actually having to put anybody in harm's way, right?
We can simulate cars running red lights so we can actually without driving a car in a dangerous situation, we simulate those hazardous situations, and we can run these simulations as a way to teach the car what to do.
Wow. Well, I cannot wait for the cars to drive me around.
I'll tell you that. So get back to work.
Danny, thank you so much. My pleasure. PostScript here.
If you are listening to this podcast on iTunes, Google Play Music, or whatever your favorite podcast player is, do us a favor and leave us a review.
Thanks for listening.
