Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz.
There are many, many things I love about living where I live in Oakland California, just across the bay from San Francisco.
The roads, however, are not one of those things.
Potholes are everywhere around here.
I'm lucky.
The street I live on was repaved a year or so ago as part of a large construction project.
But once I go down the hill from my house and make a right, it's back to Pothole City.
Potholes are annoying, and they can be dangerous to cars and bikes alike.
And they're also expensive and a pain in the neck to deal with, at least in the ways they've historically been dealt with.
But you guessed it.
My guest today is doing something about it.
Dr Benjamin Schmidt is president and co-founder of Roadbotics, a Pittsburgh, Pennsylvania-based company that's using AI to improve the world's roads in conjunction with the world's governments.
Over 160 governments across the world currently use robotics technology to fix the right roads at the right time, while saving taxpayer dollars.
And Ben's here to tell us all about it.
Dr. Schmidt, welcome and thanks for joining the NVIDIA AI podcast.
Thanks, Noah, and thanks for having us.
So just start by telling us about what robotics is and what you do.
Sure.
So we are a tech company that, as you said, solves the problem of potholes.
Infrastructure is a really big challenge for a lot of governments because of its size, its complexity and the cost.
So what we do is we help all the governments around the world to better monitor and maintain their infrastructure.
And the way that we do that is by using and combining AI with smartphones.
So basically robotics we take a smartphone, we put it up in the windshield of a car and we drive around collecting video data over the entire network.
We take that video data, we throw it into our deep learning net And that net can actually identify all the useful and indicative surface distresses on a road.
We take all that together and then we assign a score to that particular section of road and we hand that over to a government.
Now, what that means for the government and for the citizens of that government is that now they have a comprehensive and objective data set from which to start making decisions about their infrastructure.
So let me back up for a second and ask you to nerd out with me a little bit, a little like pothole 101.
What's an alligator crack?
Sure.
So yeah, there's a whole entire universe of distress types that you have to learn about.
As with everything, right?
Yeah.
So effectively you can think of a road as sort of like skin, in the way that you know what you're trying to do is prevent things from getting into the road surface.
So as soon as, and it's usually moisture is the enemy with the road.
So, You put down the base layer, you have sort of the materials in there.
Over time, it starts to dehydrate.
And also just the wear and tear of vehicles starts to press down on both the surface but then also the base layers beneath it.
As that happens, you start to see cracks forming along the surface of the road.
Those cracks eventually get wider and wider and deeper and deeper.
And in areas that get like freezes and thaws or lots of rainfall.
The moisture gets in there and it can erode both the base layer, but it can also erode the surface itself.
And this is just this is how pavement sort of operates.
Right.
So just over time, this is the inevitability.
So normally when you see sort of like cracks on a road, you know that's what's happening is they're just sort of getting towards there.
An alligator crack is so named because it looks like alligator skin, where it has like small little patches, kind of all over dotted.
And what that actually means is that the base layer underneath so not the actual piece, that is, the pavement, the asphalt on top, but rather the parts underneath have actually eroded away.
And so you can think of it as almost like when you press on a hard-boiled egg it kind of like collapses underneath but the skin stays intact, making that cracking pattern.
That's effectively what's happening.
Now what that leads to eventually is that those small little pieces of asphalt that are left there, they start to chip away and then eventually you have a pothole.
So this is very interesting because all of these distresses are things that you can monitor for.
But then you think about, you know, roads are massive, right?
There are hundreds, thousands of miles that a single government might need to manage.
And so you're trying to sort of monitor these things at a massive scale that are constantly changing and evolving.
And that's where the AI comes in.
The really interesting part about these distresses and sort of understanding them.
You know, when that crack first starts to form.
There are things you can do to really prevent the downstream bad things from happening.
It's all about preventative maintenance.
So the sooner you can detect these, the sooner you can put a cheaper intervention in now.
That really just saves the life of the road.
And that's how you'll see massive benefits in a cost system just taxpayer dollars and better understanding your roads.
So really, the idea is that Once you've seen a pothole, it's way too far gone.
That road is destroyed.
And so really, what our technology is all designed to do is to see those things well in advance, and then to intervene when necessary.
So let's dig into that then.
How does your tech work?
Yeah.
So you know, as I said, you take that smartphone, you're traveling around and collecting video data.
You can now train an AI to actually identify each one of those distresses.
So we call it painting.
We've had labelers go through and actually paint on each one of these surface distresses.
So here's an unsealed crack, here's an alligator crack, here's a pothole, lane markings.
We've had it trained with millions of those labels that then we train this deep net to actually be able to reproduce those answers.
And from that, we can then actually generate a score just as a civil engineer word.
So exactly matching today's sort of like high standard scoring systems that a civil engineer pavement engineer would be capable of doing.
So that's really like the deep core of it is this deep learning net.
And again, the neat part here is with advances in both what NVIDIA has been doing with GPUs and sort of their power, but also with sort of these advances in how convolutional neural networks work and sort of the applicability, the ability to deploy them now in production.
We couldn't have done this four or five years ago.
So it's really.
Robotics was sort of founded at exactly the right moment, with the right tech, right hardware.
And so we're now in this sweet spot where we can actually deploy a solution with machine learning, but all over the world, that actually delivers a really promising and useful product that helps everyone.
Now are you able to run the machine learning detection of the problems in the road on the edge on the smartphone itself?
Or does the raw data get uploaded to, you know, like a cloud server to crunch it?
So we upload all of it.
Now, we could run it at the edge.
But the thing with us is that, you know, having that data set.
So, you know, today we're talking about roads and pavements.
But as we continue to move forward.
You know, you imagine you're sitting in the passenger seat of a vehicle.
That's effectively what we're seeing.
So you imagine all the other kinds of infrastructure that's in that same roadway that a government's responsible for.
Same exact principles apply.
So you think about traffic lights, street lights, signage.
These are all assets that are both important for safety, just for basic navigation, that we need to know about, understand their status.
So we've been uploading all of the data for now, mostly because it goes back into our training system, so that we can start building and developing the next generation of models and really just keep advancing through that infrastructure assessment.
Eventually, it's a lot of data, so we'll start doing more and more edge processing.
But for now it's really just been.
We get to upload it to our clouds and that's where we can throw at it the really big GPUs that can really just handle much deeper nets and things like that.
So we're recording this in late November of 2019.
Back in February of this year there was an article on the NVIDIA blog which everybody should go check out.
It's a great article about what you guys were doing then and using the dash-mounted smartphones you were talking about.
But my understanding is that you've kind of opened up the data collection process to more folks since then.
Yeah.
So in our earlier days what we were doing, you know.
Your mind's naturally drawn to the AI side, which is deep and challenging.
And it requires, you know, some really sophisticated and great data science capability here in-house.
But one just really big challenge actually is the logistics, right?
Right.
So road networks are not usually grids, you know, unless you're in sort of a Chicago or downtown city.
So actually getting every single road and making sure you have a good image of every road is really a big challenge.
So in our earlier history and we're not that old to really say that we have an early and late history, we're only three years old.
But earlier, what we were doing was that we would go and make sure that we got all that data.
Since then we've built a whole bunch of turn-by-turn navigation and other aids that help a driver ensure that they got full data collection over the entire network.
And so starting really, you know, we're having our early tests now.
We're actually going to open up data collection to anyone.
So you'll be able to just download the app and start your data collection.
Begin this infrastructure.
It uploads to our servers and then poof, you get a map that has your entire infrastructure, your entire road network, assessed.
You know, just again, power of a smartphone mixed with AI.
And which is great because for somebody like me, who both always has a smartphone going for music or directions while I'm in the car, not looking at it just running in the background, maybe on the dashboard mount I can actually do something about it now by collecting data for you guys as I'm going, and then it can be put to use, which is super cool.
And that actually leads me to something you were talking about.
Obviously you're working with governments in the US and worldwide and talking about data collection and that kind of stuff.
And in certain circles and these aren't necessarily small circles these days folks hear the words AI and government and they get a little suspicious.
And if you put those words together AI in the hands of the government some people might get a little more suspicious.
You guys are working with governments.
And so I'm wondering, kind of twofold to this.
How can governments start to better employ AI or what are you doing?
What have you seen positively towards that end?
And then also, you know how can we kind of talk to the public and kind of, you know I don't know if reassure is the right word but get the public on kind of the right understanding of you know look, actually giving the government high tech tools and using them the right way can really help our day to day lives with things like fixing potholes.
Yeah, I think that's a fantastic question.
And I think, being in a space, you know, in a GovTech space, where we, you know, our partners are governments, it does present some interesting and unique challenges.
I mean governments are certainly known for having procurement sorts of difficulties, especially in a selling capacity.
But I think throwing AI into the mix in today's environment and you can definitely have a little bit more of a challenge.
I think, first and foremost, you know, one of the things that we just lead with in a lot of our, our materials, but just how we sort of present ourselves is that AI is a tool right.
It's a tool in the toolbox.
It's one thing to help sort of solve your problems, right?
And I think you know that's helped us tremendously in making sure that you know we're talking about AI not as this sort of like magic terminator like XD, but rather just something that helps right.
It's all about making this a little bit easier.
This is a very easy.
It's sort of a cut and dry example of something where everybody wants it to work right.
Everybody wants just a little bit more help in there.
Both the government.
The government is ecstatic to sort of see tools that can help streamline their processes.
But then also, as you said at the beginning, just citizens, right?
I have a bottle on my street.
I want it to be fixed.
So I think our sort of mission has helped there.
Let's make sure that every citizen has a good, safe transport.
That helps.
But I think the third is really making sure that we assuage those doubts.
So one thing certainly with a camera is always around privacy.
So we're going around, we have video data of lots of streets, but we automatically go through and we anonymize that.
So we blur over people cars, all that kind of stuff, so that you can only really see the infrastructure itself.
And that was really a critical piece to get both governments and their constituencies on board with the technology like ours, to say that it can help but it's not going to harm, it's not gonna create some sort of privacy nightmare for everybody.
But yeah, I think it's certainly been a challenge, but one that I think we're more than happy to step up to, just because of the nature of the problem in itself.
It's just everybody uses infrastructure.
It's everywhere.
We're speaking with Dr. Benjamin Schmidt.
Ben is the president and co-founder of Roadbotics, a company based out of Pittsburgh Pennsylvania, that's using AI to fix potholes, improve the world's roads and work with governments to improve infrastructure, as he was just saying, for all of our benefits.
Ben, let's switch gears a little bit and talk about your background and what led you to I guess about three years ago, when you co-founded Robotics.
How'd you get into the field of computer science, AI, and interested in roads and GovTech?
Yeah, I've had a little bit of an interesting journey through different industries, but I think it's always been themed with AI.
So I actually, my PhD is in bioengineering.
So I was doing... brain imaging at the time.
So for my graduate work, I was looking at sort of how networks in the brain evolved and processed.
And the reason that I actually got into AI originally was that this is a little bit of an interesting story.
But so that- We, like those, go for it.
Absolutely.
The school that I was working at, you know, I had to make my proposal for my dissertation.
And I found out that we actually had a dataset that had children with and without autism.
So being a good sort of machine learning advocate-
I thought, hey, this would be pretty interesting.
Take it, throw it into a machine learning model and then see if it can actually produce an answer for you.
It could label that data back and tell you, make a prediction about whether or not someone's brain scan could tell you whether or not they have autism right.
And all the attendant benefits.
You can have earlier intervention before symptoms or signs are showing.
So it seemed like a really interesting project, a neat way to sort of take everything in machine learning and deploy it at a really interesting and complex problem that could have real-world benefits.
Well, that was like, I don't know, 10 or 15 years ago.
Grad schools at the time still hadn't quite come to grips with the entire machine learning world.
And, quite honestly, you know, the world of machine learning tools was nowhere near what it is today, with deep nets and massive GPUs.
I mean, I remember thinking about like a few megabytes of processing on the GPUs back then.
It's like, okay.
That was really my first foray.
I did some more work on that in grad school in different ways.
But I think that project really stood out with me in terms of something that could have a real world benefit if it works.
It certainly could maybe not have worked, but I think that's really interesting.
I left grad school, actually went to another company that was doing power forecasting.
So we were looking at what the price of power looked like over the next few hours.
And again we were using all sorts of machine learning tools to sort of predict those a lot of the people at that company.
So that was another startup here in Pittsburgh.
Just to back up for a second.
Sorry.
Did you do your, uh, undergrad graduate work in Pittsburgh?
I did.
Yeah.
At Pitt at the university of Pittsburgh.
Great.
Yep.
Yeah.
That's so.
I grew up in New York, came out to go to Pitt for undergrad, stayed for grad school and then just kept staying.
So I've got now memories of watching big East basketball on TV as a kid going through my head, but that's another conversation for another time.
So, yeah, you know, they always get close, but you know, right.
But yeah, so I ended up at that power company, again using machine learning, but this time for like, time series forecasting.
And at that company, I actually found my co-founders for this company.
So Robotics is at Carnegie Mellon University, Spinout.
My co-founder, Mark DeSantis, he's been an adjunct professor at CMU for a number of years.
My other co-founder, Christoph Mertz, has been with CMU for a number of years and he's really the visionary.
It was his brainchild, this idea that a smartphone is this incredible sensor device that's in everybody's pocket.
He actually worked with some of the original autonomous vehicles when he was with the DARPA project.
Oh, very cool.
And so his idea was like okay, how can we merge a smartphone device And think not about the vehicle itself, but rather everything else, right?
All the things the vehicle needs, like the roads.
So I think with this idea of Christoph's, we really thought there's a market here.
There's an opportunity.
I think the technology certainly existed.
Christoph had proven that part.
And so then we set to work sort of trying to understand the business model.
You know, how would we get this out there?
What would be the challenges associated with it?
And that's really the start of Robotics.
We licensed some of the work from Carnegie Mellon.
And yeah, we're coming up almost on three years now.
And so kind of looking ahead.
You mentioned the app coming out sometime, possibly Q1 of 2020, that will let you know folks, as they're driving around record video to send to help the cause.
What else is in the future, you know, both near term or further out for Robotics?
Very good question.
So I think certainly you're going to see sort of two parallel paths.
You know nothing radical in it.
One is just as I was sort of alluding to before.
There's a lot more that you can see in that same image, that same video data.
So lots of other infrastructure that's monitored by governments has the same exact challenges.
You know, it has to be maintained.
You have to understand its condition.
It It's very costly.
Like all infrastructure is just a continuing process of maintenance cycles.
So that's one.
I think deploying more and better AI towards understanding that built world environment.
That's going to be one area.
And that is now imagine we're 10 years from now.
And you're a government who now has you know every quarter you've been collecting and updating your infrastructure data right.
You've got 40 of these assessments across hundreds of miles potentially.
And you're going to look at things or you're going to want to learn about things like how fast are the roads degrading?
Why is this road degrading faster than that one?
What's the relationship between crash data and road condition data?
If I can see the stop sign from 100 meters away, does that reduce the number of crashes in that intersection?
Does that change the way the road conditions look?
What about how do lane markings sort of affect traffic?
You know, maybe property values, right?
There's a lot of this sort of interesting next level data that we'll be able to have because we have just such a comprehensive understanding of the physical environment of that community.
And that's going to be really, really exciting is how do you manage that?
How do you extract information from it?
And ultimately just, you know, build better cities for everybody.
That's going to be really exciting.
You know, it's interesting.
As you were talking about that, I was thinking that not the road I alluded to at the outset, but the road just beyond that connects my little neighborhood to kind of a main thoroughfare that goes from one freeway at the top of the hill to a freeway at the bottom of the hill and then down into downtown Oakland.
And that road is in dire, I'll say, need of repavement.
But as part of that, the city is doing an assessment to see if they want to actually put the road on something called a road diet.
So reducing it from a four lane two in each direction road to one in each direction with a wider shoulder for bike lane, but then also talking about reducing speed, because people tend to drive well over the speed limit on this road.
There's a school and a crosswalk and just dealing with all of these different factors that come in.
And so I'm just imagining man, if they had these layers of data collected over the years, this rich data, but then also being able to be aggregated and crunched by a sufficient ML system.
And then, let alone if, then not only informing those decisions, but informing the people, like the ones who started arguing on the neighborhood listserv about whether or not a road diet is a good idea or not.
And you kind of realize like well, there's so much that I don't know about what a road diet actually does.
How can I even know which side to complain about?
Absolutely.
Absolutely.
Yeah.
And I think that's one of the most interesting parts for us is, You know, the first question that we have to wrestle with with every government is where are your roads right?
And it almost seems like a joke.
Like, what do you mean you don't know what your roads are?
But you know, what we mean is really a digital record of your entire road network, which very few communities really have, sort of very clear and very updated.
But then you think about, like, how does that happen, right?
Like okay well, you know you take that road, put it on its road diet, change some information about it.
That's just one project.
But then you'd have like a road nearby that might have another project three months later.
And so like, the neat part about all of this infrastructure is that it's just continuously being changed rebuilt, revitalized.
It's hard to keep track of.
And it's hard to sort of just manage and understand the current status of it.
And then you take it to that next level, sort of putting that in a format where, as you said, you can start to do the assessment, the understanding, like what's the impact of this action on this particular piece?
And so I think that's kind of the.
That'll be the neat sort of next generation theme of robotics is, you know, today we'll tell you what the status of your infrastructure is, but tomorrow we're going to help you build that infrastructure, and build it smarter and better and faster.
And that's really kind of I mean, that's just wild.
That's beyond what we originally even thought we were going to be doing.
Well as an AI geek and a citizen of a lovely place to live that's marked with potholes, I'm excited on multiple fronts.
I can't wait to see where Robotics goes.
For folks who would like to learn more about what you guys are doing, where can they go online?
Robotics.com is our website.
We would encourage everyone to talk to their local elected officials and DPWs and things about how we might be able to help your community.
So roadbotics.com.
Excellent.
Ben, Dr. Schmidt, this has been great.
Thank you so much for coming on the podcast.
And I think it goes without saying, but all the best of luck to you and the Roadbotics crew and all the governments you're working with.
Well, thanks.
Thanks, Noah.
Thanks for having us.
Thank you.
