Welcome to NVIDIA's AI podcast. There is a school of thought that the idea of driverless or autonomous vehicles will apply to anything that moves.
So that includes anything that drives on a road, soars through the sky, or maybe swims or crawls beneath the water.
Our guest, Kevin Peterson, one of the co-founders and the software lead of robotics company Marble, is certainly working hard to make that true.
Peterson and the Marble Gang are designing and deploying robots for what they call the sort of last mile of logistics.
But, you know, essentially it's delivering food, medicine, you name it, with these autonomous robots in our cities.
And they're deploying them as we speak in San Francisco and testing them out.
Kevin, welcome to the podcast. Thanks for having me.
You describe what Marble's doing, and then I want to get into how you and your fellow co-founders even came to this robotics company idea. and what your experience is.
Cool, yeah. So Marvel is... a robotics company.
We're working on last mile delivery, which means that we're working on essentially the last actually few miles of getting you your things.
So if somebody ships a package, It might end up on a plane and then on a truck and then it gets to a distribution center.
And then from that distribution center, it travels out into a city.
And that last portion of driving is where a lot of the effort is.
And so we're trying to automate the last little bit to make the system more efficient.
That last portion is our FedEx trucks and UPS and postal carriers, for that matter.
Now, I've seen photos of your robot. It looks to me like a sort of small version of a Zamboni, if you know what that is.
But how do you describe it? What does the thing look like?
How big is it? And how does it work? What we've built at this point isn't a polished vehicle.
It's a functional vehicle and so it's not beautiful yet, but it's A four-wheel vehicle looks a little bit like a car.
It's a little boxy, it's white, and it's got An area inside of it where it carries packages and it's got a big door on the side of it that looks kind of like a car trunk.
So you... code in and then open up that door and you can get your things out.
So if I ordered a meal or I'm waiting for a package, I get the code and Only I get to get in there and get the stuff out.
That's right. And it's locked while it drives.
And size-wise, it can fit on a sidewalk.
So how big is it dimension-wise? It's about the size of a mobility scooter.
Ah, okay. So the rascal that you might sometimes see at stores or whatever.
I see. So a can, it's designed to go on a sidewalk if need be, or is that its main path that it travels?
The main path is on the sidewalk, although obviously it has to enter the road to get across intersections.
Got it. So I know you're testing them in San Francisco or piloting right now. who you're piloting with and what are your robots doing?
So we just recently announced a pilot that we're doing with Yelp Eat24.
YelpEat24 delivers meals. So in the mission in San Francisco right now, you can go and order a meal from one of the restaurants that we have active.
And if we're operating a shift at that time, then the robot will deliver your food. the mission district for people that don't know is a very um i don't know dense neighborhood in San Francisco that's filled with bicycles and cars and people walking and not to mention filled with restaurants and other kinds of places so it's a very you know, dense urban setting.
So what are some of the problems or the issues that you have to face and that you guys have to tackle?
We tackle a lot of the same problems that self-driving cars tackle, except that our vehicles are operating where it's actually in a speed regime where it's much safer for them to operate.
We're going about walking speed, so that might be three, four miles an hour.
And that means that we can stop quickly and safely.
We're still testing out the autonomy. We're looking at what works well and what doesn't.
And we're trying to be incredibly, incredibly safe while we do that.
So if you look at self-driving cars for instance, There's typically two people in those cars.
Right. Not to mention the autonomous system.
So, you know, as it's driving, they're also monitoring all the driving that's going on.
What is it like for people when these things are going down the street or the sidewalk, I should say?
What are the reaction of people and how do they kind of engage with them?
For the most part, people love them. I'd say You know, kids especially go absolutely bonkers when they see the robots, which is pretty entertaining.
A lot of people... don't really interact with the robots.
So they'll walk by and they won't really even look at the robots.
A good number of people are on their phones, maybe don't even see it.
Most of the people that we interact with really like it.
They feel like it's the future, like they're living in the future all of a sudden. because these robots are moving past them.
Right. I can imagine in some scenarios that the deliveries will take twice as long just because people will want to stop and ask you or it's, you know, the robots mind your questions or just take a picture with it.
So give us your background. I know you guys all were at Carnegie Mellon in the robotics department, I would imagine.
But how did you get from whatever it is that you were studying to this idea of And then I wanna dig into how you guys are pulling it off.
Like what's at work here beneath the robot skin and all the, Kids who are laughing and screaming as they see it go by.
Matt, the CEO at Marvel, and I met at Carnegie Mellon, working on the DARPA Grand Challenge, which was the competition to build the first self-driving car.
I was leading one of those teams and he was working on that team with me.
So we built these desert racers that drove through the Nevada desert at 30, 40 miles an hour completely autonomously.
And that was a little over a decade ago now.
Tell us what it was like... a decade ago when you were working on it, like how did these vehicles operate and how that's changed today?
Because I recall For example, that was the desert race series, but there was a city series.
You know, these driverless cars would go 50 feet and then drive into a wall or just stop or whatever, you know, like.
Well, yeah, so there were a lot of teams in that challenge.
Our team at Carnegie Mellon actually did pretty well and we got the car working and finished that course without any major issues.
There were a couple little bugs that came up.
But yeah, there were lots of vehicles that had accidents.
And of course, that's just early development, right?
That was the very early stage of autonomous vehicles.
We didn't as a community understand how to make the vehicles see everything around them.
We were inventing the motion planning algorithms to make them efficient, to make them operate around moving vehicles and traffic and things like that.
And so there was a lot of invention that went into those vehicles.
A few things have happened that make it much more straightforward to build an autonomous vehicle these days.
And I would say those key technologies are now making it viable for a lot of different.
Those technologies are things like computer vision algorithms that allow us to detect and understand what the different objects are around us.
So neural networks, for instance, that will tell us that we're looking at a person or we're looking at a dog or we're looking at the sidewalk. and allow you to draw a box around those things.
10 years ago, those algorithms really didn't work.
And now we're at a point where in some cases those algorithms actually beat human performance again you're not piloting a car through the desert at 40 miles an hour but a uh you know a four-wheeled robot on the sidewalk at three or four miles per hour What are you training your system on?
What kind of data? And what are you teaching it to do?
Is it acting as sort of... a pedestrian or pushing a cart down a sidewalk or like what does it know and how does it know it?
Yeah. So we have, cameras and lasers on board, as well as sonar, those sensors give us a really comprehensive view of everything around us.
So the laser gives us the location of all of the objects in the world that we can see And then the camera tells us what we're looking at.
So it tells us the difference between a person, a dog, a sidewalk.
So there's two things that we try to learn.
One is that classification. So we feed the neural network system a ton of labeled data.
And that labeled data is basically images with boxes drawn around the objects that you care about or in the case of sidewalks and things that aren't easily boxed in.
You actually highlight which pixels in the image are the sidewalk or are not the sidewalk.
So we make a huge data set, millions of images, and then we feed that to a big neural network training system and then that gives us, that neural network then has the ability, learns the ability to classify those objects.
But that's only about half of the problem.
The other part of it is processing the laser data. figuring out where the ground plane is, figuring out where the vehicle is in the world, so localizing it off of a map.
And that's really not a learning system, that's more of traditional robotics.
When you get an order, and let's be specific about this, let's say it's a I mean, what kind of food are you carrying around the mission these days?
We are carrying some burritos. Okay. So I'll tell you, I want a carne asada burrito with pinto beans, cheese, and cilantro.
Do you map the best route? Because I'm just imagining like in the mission, for example, Yes, you're supposed to have curb cuts, but they're not everywhere.
And some of the curbs are eight inches and some are 10 inches.
Do you map the most efficient route ahead of time and then go out and make it happen?
Or does it kind of go on its own and figure it out as it goes?
It's kind of a combination of the two. There's some things that you can map easily, right?
Like where all the buildings are and the curb ramps. the addresses, right?
Things that are static and don't change.
So we do map those and we map them in pretty detailed three-dimensional maps that are super cool. before it drives, we have a system that plans a route in that map.
And that looks an awful lot like Google Maps, but it's It's our own system.
And we built that system because nobody else really has a sidewalk directions for a robot mapping system.
So that takes care of the static information.
But then as you drive, you might interact with traffic.
You might come across street fare. that blocks your way.
Somebody might have put a bunch of trash cans out because it's trash day, right?
And so we have to detect those things online and route around them.
So there's a system on the robot that detects those things and requests a reroute.
Right. So it's a combination, like you say, of plan and then, like most of us, okay, the street's blocked off or there's a big potted plant in my way, I'll go around it.
Yep, that's exactly right. Interesting. Did you, Kevin...
Were you, before you went to Carnegie Mellon, were you one of those people who were taking apart cars and building robots on your own?
How did you get to this as a... focal point for what you like to do and what you like to work on?
I have always been interested in robotics, I wasn't always into cars, but was always very into robotics, took programming classes when I was in undergrad and high school and even before then.
And then I got into college and Red Whitaker was looking at doing these Star Trek Grand Challenges.
And he put a poster up on the wall that said...
It was like a picture of a dune buggy flying through the air and had lasers on it.
And it said... Come build the future or something like that.
There's no way that I could resist that.
Yeah, I want to go do it right now. Well, are you amazed at how – and what I'm amazed by, like, you know, from you and that poster, but how – the future is kind of arriving, right?
I mean, how much better we are at this, but how much farther do we have to go and whether some of the issues that maybe involve technology, but also involve cultural barriers, if you will, or regulatory, or what else do we need to think about?
Yeah, the regulatory one's an interesting one, although I think it's being solved.
I think as we see solutions come out, regulators will work with us to make things happen, work with the automotive companies to make sure that cars get on the road.
And part of that is really that there's a huge amount of impact in the world that comes from having these kinds of autonomous vehicles out there.
Yeah, I want to ask you, I mean, we sort of talk about this as if it is a foregone conclusion that this is the future, but why should it be as opposed to why? you know, will it be?
Yeah. And there are a lot of really good reasons for that with cars.
The obvious one is accidents, right? There are thousands of deaths every year in the United States alone. from cars that crash.
But if you look at the amount of time that you save, if you never had to park again, The numbers get really crazy.
So something like 30% of the cars on the road are parking at any given time.
Right. Just circling, looking for a parking spot.
Yeah. Those kinds of numbers are true in the rest of robotics as well.
So do you have any idea how many minutes a day the world spends going to the grocery store?
I have no idea, but I'm sure you do. It's something like 30 billion minutes a day, right?
30 billion minutes a day? 30 billion minutes a day.
And that's across the whole world, right?
Right. robots aren't immediately going to be everywhere.
But can you just put that into some perspective?
Do you know what 30 billion minutes a day adds up to in months and years?
Quickly do the math. So that's about 20 million days every day that the world gets back just going to the grocery store.
Oh man, yeah. And that doesn't even include shopping.
That's just driving to the grocery store and driving back.
And so if you look at all of the different things that people are doing and the amount of time that they're spending on those things.
And imagine that instead of taking 20 million days and going to the grocery store, the world took 20 million days and tried to cure cancer, right?
What's the impact of those hours? And of course we don't know what that will turn into.
We didn't know what the agricultural revolution would turn into.
But robotics has that same impact. And one of our goals at Marvel is to address problems that take on those sort of magnitude impact problems.
Right. How do you then answer the question of like well but what about all those you know delivery people what are they going to do with their time or you know because they're going to have a lot of it if they don't have jobs There's some interesting aspects there.
One is that Amazon, for instance, has found that as they've introduced automation into their supply centers, They've actually grown the number of people that are involved, which is unintuitive.
But what happened is that as their services become cheaper and cheaper, the demand grows, right?
And so... It actually ends up increasing the volume so much that it increases the number of jobs.
I don't know if that's true across the board.
I'm not an economist. What I do know is that robotics is the future across the world, and it's going to increase the speed of our economy and increase the productivity of our economy.
And so in the long term, it's where we're going to end up.
And we need to be working on these problems to be able to move into the future.
For those people who want to work on these kinds of problems, what is the sort of mindset or a skill set that you need to bring to the table.
And I mean, we can't all go to Carnegie Mellon, but Is there more and more opportunity, do you feel, for people to take part in this revolution, like you say?
Yeah, I personally believe that motivation is the number one factor behind being able to learn these things and robotics because things are moving in the world.
You can program something and you can see it move around.
That is very motivating. I've seen lots of kids get excited by that.
There are many different disciplines that go into robotics.
I've studied a lot of math. I'm a computer scientist and an electrical engineer.
That's the software side, but there's also building the vehicles, We have just as many hardware engineers here as we do software engineers.
There are business aspects to it. These are domains that Nobody else has looked at yet.
And so there's interesting new businesses to be formed.
What do you know? Can you give us an example of domains that are out there that are ripe for good ideas?
I want somebody to invent a machine that'll fold my laundry.
I'm with you there. Yes, absolutely. Yeah, and if I knew how to do that.
Everybody would want that. It turns out that's a very hard problem because laundry, a piece of clothes, you pick it up.
A computer has a really tough time figuring out what this soft floppy material is and figuring out that it's your shirt or your underwear.
And so it turns out that's an interesting both manipulation problem and computer vision problem. no I like that are there other areas that you see out there that people are working on that you find fascinating the way to think about it is anything that's that's incredibly routine could be automated.
So if there's a task that many, many people are doing over and over again or You know, the typical phrase is dull, dirty, and dangerous.
So tasks where there's radiation, tasks where you have to go into an area that's not conducive to human life.
So underwater, in mines, things like that.
I think the ones that touch us every day are things like cleaning floors, driving is getting close to done.
It will get done over the next five years.
Any of these tasks where it's routine, it doesn't take a lot of creativity.
You mentioned that within five years the driving problem will be solved.
Spin us forward five years and include driving, but include other robotic tasks.
What is that same trip down a... a sidewalk in the Mission District in San Francisco start to look like?
You'll see vehicles like ours on the sidewalk that are delivering things, cleaning things up.
I could imagine trash vehicles. out there sweeping the streets.
I don't think it's going to feel so crazy.
I think it's we're gonna think of these vehicles kind of like your toaster, right?
It's just a thing that's doing things for you.
It's like your washing machine. Before you had a washing machine, You had to wash all your clothes and it was this incredibly labor intensive task.
Now we have a washing machine in the closet and just washes things.
And I think that that is the inevitable conclusion.
The vehicle that you guys have out there is in many cases one of the first examples of an autonomous vehicle that people will interact with.
Do you guys have a particular responsibility?
I mean, do you think about that in any...
Oh my gosh, we have to make sure that everyone's first interaction with a robot should be like X or Y. And how do you view that?
And what are you doing about it? We want everyone's first interaction with the robot to be delightful.
Actually, so we spend a lot of time designing that interaction and making sure that the vehicle is operating in a way that looks good and is good.
So take us through the real interaction.
Like I order my food, do I get to watch its, progress?
Do I get a song and a dance when it arrives or what happens?
Yeah, so you order just the normal way that you would order through Yelp.
And then the robot starts driving and Yelp actually doesn't have a way of telling you telling their customers where the robot is, but in the future we plan to roll that out.
The robot gets to its destination and then if you order that food, you'll get a code and you type that code into the door, the robot makes some noises and and then the door opens and you can get your food out.
Do you guys have a name for it? Is it got a nickname or is it just the robot?
We call the vehicles that we have right now Happy.
Happy. Okay. I will keep an eye out for Happy in the Mission District.
In fact, I will order some food to make sure Happy can show up in my neck of the woods.
Kevin Peterson, thank you so much for joining us.
Thanks for having me. PostScript here. If you are listening to this podcast on iTunes, Google Play Music, or whatever your favorite podcast player is, do us a favor and leave us a review thanks for listening
