Hello and welcome to the NVIDIA AI podcast.
I'm your host, Noah Kravitz. The path to self-driving vehicles is usually marked by six milestones, or levels as they're called.
Level zero is full human control. There's no automation involved.
Level five is full automation, aka the mythical self-driven car.
Right now, in April of 2019, you can buy a car with level 2 or level 3 autonomy, but those still require a good deal of human piloting and intervention.
Jingxiang Xiao has his sights set on level four, defined by the National Highway Traffic Safety Administration as a car capable of performing all driving functions under certain conditions. conditions.
Does that mean we can take a nap or read the paper while our cars drive us to work?
And when can we have these cars anyway? Jingshang is here to shed some light on the state of self-driving vehicles.
He's the founder and CEO of AutoX, and he's recognized as a pioneer in the fields of 3D deep learning, RGBD recognition and mapping, big data, large-scale crowdsourcing, and deep learning for robotics.
He gave a talk here at GTC 2019 entitled Enabling Level 4 Autonomous Driving Technologies, in which he explained how AutoX uses cameras, LiDAR, sensor fusion, large-scale high-definition 3D mapping, and simulation to make ultra-robust autonomous vehicle software and hardware. and also how their self-driving technology is changing retail, mobility, and other industries.
That's a mouthful, so let me let the professor himself explain it.
Jingcheng Xiao, thank you, and welcome to the NVIDIA AI Podcast.
Thank you, Nat. So first of all, offline, you told me, and I called you the professor, your nickname, Professor X?
Yes. Fantastic. I love it. I may defer to calling you Professor X during the podcast.
I hope that's okay. So I kind of briefly tried to outline, and there was a lot in that intro, I know, but your background, you know, you've touched on so many areas and the technology that you're working on now is...
Such a big part of what people these days think of when they think of autonomy and AI, self-driving cars is one of the big things. but then it has applications in other industries as well.
So there's a lot in there, a lot to get to.
But let's start with this idea of levels of autonomy for vehicles.
Could you maybe explain for the listeners what level zero through five really mean and where the industry's at right now?
And then from there, we can get into what AutoX is doing.
Certainly. is very simple. That means there's no autonomy.
The car is fully manual, driving fully by a driver.
A human driver. Right. A level five, as you explain, is fully automatic.
It's just like you hire a human driver to drive the car for you.
Right. A level one, a level two is like some kind of assisting driving.
For example, a Tesla with autopilot is really a level two feature.
They can steer in. do the steering wheel control.
They can control the brake, control the throttle on the... usually on a highway scenario, doing land following.
But Global Street is a very interesting aspect because the computer can drive the car by itself.
But you still have to have some kind of supervision.
Usually that means the computer or the car may give you some notification.
If the car feels uncomfortable or being able to, unconfident of being able to drive the car safely, they will give you some notification like, 10 seconds before the problem.
Okay. So that you have 10 seconds ready to react.
So you can react, you can take control. Yeah.
Right. Exactly. But that means you will not be able to go to sleep.
You can check your phone for a text message, but you will never go to sleep.
Right, right, right. And level four is the step up, that complete autonomy.
That means you can actually go to sleep.
It may mean that a car may have nobody. sitting inside a car, it's a completely empty car, that's level four.
And how does that differ from five then?
Level 4 usually is constrained by ODD operation domain.
That means it is Under certain area, usually it's a geofence area.
Got it. Under certain condition, if it's a huge earthquake situation, Then the car probably will not be able to operate.
Right. situation so it has a constraint level 5 means there's absolutely no constraint as long as a human can drive a very skillful human driver can drive, the machine should be able to drive.
Got it. Right now in the industry, I mentioned Tesla managed to get level two working very well.
A few automakers are also pioneering level 3.
There may be a certain limitation of the level 3 now. but still there are some level 3 features available on the market.
But for level 4, A couple of companies are working on it, such as the most famous one, the Google self-driving car.
And GM has GM Cruise. And Apple, people wait for news, have the secret special project in self-driving car.
Right. AutoX is also in level four. So right now, almost all the top tier self-driving car company, are spending a lot of time and effort on doing research and development in level four.
Level five is still a fantasy. We basically have to solve the general artificial intelligence in order to get to level five.
That's why basically everyone is focused on level four now.
Right, right. And so you mentioned AutoX is working on level four right now.
What exactly does AutoX do? AutoX is a technology company.
We focus on developing the AI driver for level four self-driving platform.
So we are technology-enabled. We're mostly building the software to provide a very good AI brand to power the self-driving stack.
Of course, we do a little bit of hardware Because sometimes we have to.
But we take most of the hardware off the shelf.
For example, we don't build a car. We don't build a LiDAR.
We don't build a camera. We don't build a computer chip.
Like what? NVIDIA, right? But when you make use of the very good hardware ecosystem and the general automotive ecosystem, But we're filling the missing gap, which is the AI software to drive the car.
This is a big, broad question, so feel free to narrow your answer down on the relevant parts.
I can only imagine all of the different systems, all the different sensors, all the different data, all the different decisions that have to be made in an AI driver.
What are some of the biggest challenges that you're working on right now when it comes to trying to build that level four AI driver?
Yeah, because we started two years ago, we have spent quite some time for R&D.
So we have basically finished all the basic stuff that we have been doing. driving our car on the road since two years ago.
We have accumulated a lot of mileage in terms of testing.
So right now, our primary focus is on two aspects.
One thing is on scalability. We want to mention level four is always geofence in a certain area.
So we want to extend the area to be bigger and bigger.
Right now we have been operating mainly in San Jose area.
But our hope is to extend bigger to other cities as well as like covering bigger area in the San Francisco Bay Area.
Why does the geofencing come into play? Is it so that the vehicle can learn the terrain, or is it to control what types of other vehicles are on the road?
Is it regulatory? It's not really regulatory because according to the California DMV, regulation, we can actually drive inside California almost all the public roads.
It's really because of the technology. The technology depends on the high definition HD maps.
Because it's unlike a regular navigation map that just tell you general direction.
Right. HDMap tells you very, very detailed features like buildings, street signs, traffic lights, traffic lanes, all kinds of by detail information up to a centimeter accuracy.
Why does a self-driving car need this map?
Because this map provides a very rich source of information for the self-driving car AI to make use of.
Certainly we can build an AI that drives a car without HDMAT that would have been done like two years ago.
But to make sure it's 100% safe. Yeah. This is really another layer of protection.
While the car has an onboard AI running in real time, at the same time, we also use the HD map. as a prior knowledge, combining these two kinds of knowledge, the prior one and the current one, together to ensure the safety.
That's why almost all the levels for self-driving car companies are making good use of the HD maps.
And HD maps, by definition, is constrained by a certain area.
If there's an area has HD maps, You can drive, if there's area without HD map, you cannot drive, self-driving.
Simple enough, yeah. Yeah. And the second aspect I mentioned is really about polishing the software to fix the corner cases.
We all know that self-driving cars are difficult because there are always some random corner cases that make it very, very difficult.
So Waymo have been spending a lot of years, many years on polishing their software.
Similar here, we have been working very hard to catching up. as well as producing our software.
Yeah, and this is more about the technical side.
On the business side, we're really looking for the best use case for autonomous driving.
A lot of people may feel that autonomous driving is cool, but especially level four, it's very, very cool because the car indeed is driving itself.
Sure. But it's actually... so dramatic change to the whole industry that how we make good use of the level fork vehicle.
For example, a lot of people feel that because it's a new technology, the general public may not have enough trust on the new technology yet.
It's almost true for any new technology.
It's just an airplane. So finding a good use case for this new technology is also a very important task.
The same technology can actually be used for other stuff, such as food and grocery delivery.
We feel that it is probably easier for people to accept this technology if you are not sitting in the car.
You're not in the car. Sure. That makes sense.
That resistance to new technology, that's a universal thing. finding the right in can make all the difference.
Can you speak a little bit to the advances and should mention before AutoX, you mentioned you're a professor and you've got a... storied background working with a lot of different technologies in this field.
How have some of the more recent advances in AI-related tech, both on the hardware and software side.
How have they kind of accelerated the growth of self-driving cars and auto access work?
Yeah, certainly. I got my PhD from MIT and then I actually been working in self-driving car for more than 12 years now, two years ago.
When I was still in college, I started to work on self-driving car.
At that time, our project is actually founded by Google.
Google founded the university with some money and data.
And then we started working on that. Because at that time, it was not that many people working in self-driving cars.
Right, so this is like 06, 07? Yeah, around that time.
Yeah. Yeah, exactly. So since then, I've been working on that and we have been...
While I'm doing my PhD at MIT, we're working with Nissan from Japan, founding our lab to do some self-driving car project.
And after that, I graduated from MIT. I went to Princeton University to become a professor.
I was a professor at Princeton for three years.
I was the founder and director of the Computer Vision Robotic Apps.
So we have been working on many things, including self-driving cars and 3D deep learning.
In particular, deep learning is a revolution for self-driving cars.
Of course, for the general AI as well. Why cars in particular?
Because self-driving car, the most important thing is to recognize the objects.
If you cannot recognize a pedestrian, that's a huge problem.
That's a problem. But getting the performance to be very, very good because for self-driving car, compared to the rest of AI is special in the sense that you cannot make a single mistake.
Everything has to be perfect. So the requirement is much higher than a lot of other AI applications.
That's why we are so thirsty for performance.
But before deep learning, the performance or object recognition is always not that great.
It's working, but it's not good enough for self-driving car applications.
That's why... The deep learning is really giving a big boost to the self-driving car industry.
Around 2012, everyone knows that the deep learning revolution finally took off.
Around that time, deep learning is usually processing one-dimensional signals such as natural language processing or two-dimensional signals. such as image and video.
But self-driving car is different because Everything is in 3D.
You need to recognize the object in 3D. You need to recognize the projection in 3D.
And the signal... Data signal, for example, if I use a LiDAR, a laser scanner to scan the environment, the data comes in three dimensions as well.
Sure, yeah. So at that time, I look at all these applications and realize maybe we can extend the deep learning technology from 1D, 2D to 3D. and use it for self-driving car.
That's why I start to work on something called 3D deep learning.
It's a term actually coined by me because...
Because at that time everyone is focused on 1D and 2D.
And I was the first to extend it to 3D. And we use convolutional neural network in 3D to recognize the object truly in three dimensions.
That's actually a very important part of self-driving car today.
Today, almost all the level four self-driving car system are using 3D deep learning kind of technology for the object recognition.
They take the LiDAR data as input, they use the 3D point cloud from the LiDAR, and they run a deep learning network on these LiDARs, a LiDAR point cloud.
Yeah, that significantly increased the performance of object recognition and that significantly advanced the self-driving car space.
We're speaking with Zhengzhong Zhao. He is the founder and CEO of AutoX.
And as he was talking about, he's got a background in... all manner of things.
He's been working on self-driving cars for a while, but has a deep background in all manner of technologies related to deep learning and also robotics.
We said at the outset that some of the stuff that AutoX is working on also has implications for other industries, retail, mobility in particular.
Can you speak a little bit to that? Certainly, autonomous driving is a new technology.
A lot of people may feel, how do I use this technology in our daily life?
One thing that particularly strikes us very interestingly is like the... food and grocery delivery.
Because as we know, people in U.S. already in houses and those houses are usually very far away from restaurants.
Right. Full delivery, for example, is already a huge industry.
But right now, because the labor cost is very high, For example, if you spend $10 to buy a pizza, you may also actually spend $15 to get the pizza delivered to your house.
Right, exactly. It's really not that convenient.
Yeah, that's why this self-driving car technology can come into play.
It can reduce the cost dramatically. Now you are not having a human doing the delivery, but actually have the machine to deliver.
Beyond the cost, there are a lot of other benefits.
For example, privacy. For example, if you order a pizza at midnight, you probably don't want a random stranger to show up at your house.
Right. This calls to mind, you know, delivery bots that are working on Last Mile delivery had on the podcast, well, earlier at GTC, Somebody who's doing a trial, their company's doing a trial at Cal Berkeley delivering food, but they have kind of a hybrid method where a human on a tricycle is driving, riding around with the delivery bots, working with the restaurants, getting them loaded up, and then sending them off.
When you're talking about self-driving cars, are you extending that to think about these delivery bots and such?
Or are you really kind of thinking more about what comes to mind when we say car?
Yeah. That's a very good question. It really depends on the distance.
In downtown, like San Francisco downtown or New York downtown, This cloud, a small robot running on the sidewalk will actually work very well because it reduces the traffic. because the distance is small.
But I would say 95% of U.S., Everybody in house in suburban area is actually too far away to have a delivery robot just running on the sidewalk to do delivery.
So this is where the self-driving car can come to hell.
So we believe that in self-driving car, except the downtown area of most cities.
Or the other area is more suitable to have a self-driving car to do delivery.
Almost any kind, not just food and grocery, but also other packaged delivery. is also more suitable using a regular car.
You know, I grew up in the suburbs myself and I feel I feel almost guilty having asked that question, heard your answer.
I've lived my adult life in cities, forget so quickly that yes, most of the United States is spread out and it's not suitable, but it's easy to get caught up in our bubbles.
So- As much as you can say, because I'm sure there are things you're working on and there are business aspects of the operation that are, you know... stealth, so to speak, right now.
But what's next for AutoX? How far off do you think we are realistically from level four vehicles and whether they're robot taxis and automated delivery vehicles or... or cars for personal use.
How far off do you think we are from these actually hitting the roads and, you know, some sort of quantities.
And then also to go back to it, what's kind of in the future for AutoX?
I think we're actually very close to this future as the technology, the industry grows, not just AutoX, but also a few top players. in this area have been polishing the technology really quickly.
If you look at the 2018 California DMV report for autonomous driving.
I would say the top 10 players in terms of the number of mileage, they're actually doing very, very good.
So I really see that we're making a huge progress since I have been working in self-driving car 12 years ago.
It's a huge, significant progress. We are very close.
I would say that in a very near term, probably in two or three years, people, the general public will see that there are some self-driving cars Probably not everywhere, but a certain area that start operating unmanned without any driver inside.
This is very close. That's why we're also very interested in this aspect to really push the technology further because we're just right behind the field.
It's a brave new world, but it's coming upon us fast for sure.
Xingcheng Zhao, thank you so much for coming on the podcast.
There's so much more we could get into, so we'd love to have you back sometime to talk about other aspects of this work.
Certainly interested in your work at Princeton at the Computer Vision and Robotics Labs, but we'll save that for another time.
Hopefully you'll join us again. Thank you very much.
Thank you.
