Welcome to the AI Podcast. I'm your host, Katie Burke Washabaugh, covering all things autonomous. both in the real world and the virtual one.
In an earlier episode, NVIDIA's Clement Farabay walked us through the AI Data Factory needed to develop and train self-driving neural networks at scale.
The next step in this process is testing and validation, which requires high fidelity simulation.
And like much of the technology related to AI, simulation is constantly evolving and improving, getting ever closer to closing the gap between the real and virtual worlds.
It's this real world accuracy, or what we refer to as fidelity, that's critical to testing and validating AVs in a way that will translate to physical roads.
Joining me today is Matt Cragen, Senior Product Manager for Autonomous Vehicle Simulation at NVIDIA.
With more than 10 years of experience in robotics and automotive, Matt has been on the ground floor of the development of the NVIDIA DriveSim platform for autonomous vehicles.
Hello Matt, welcome to the podcast. How are you today?
Good, thanks for having me. We're thrilled to have you.
Simulation is such a blossoming technology in the field of autonomous vehicles and I have so many questions about recent developments and the DriveSIM platform itself.
So let's dive right in, get started. How did you yourself get into automotive and specifically simulation?
Yeah, great question. So my background is actually not computer science.
My background is actually mechanical and aerospace engineering. and I started out my career doing computational fluid dynamics. basically virtual wind tunnels.
And we did a lot of work in automotive industry.
So a lot of work for OEM automotive companies and a lot of work in racing.
And later on in my career, I transitioned to Nvidia.
And when I came here, I really still had a love for simulation and for a lot of the work that was going on and was really excited about autonomous vehicles and it was a great entry point. the things that are actually being developed and tested are slightly different in terms of the external surfaces of a vehicle or an auto versus the AI algorithms that are used to drive an autonomous vehicle.
Those are certainly different things, but a lot of the challenges in simulation are very much the same.
There's a lot of challenges associated with simulation in terms of how do you get to the point where you have confidence in a simulation to make a decision?
How do you know that the results that you're getting out of your simulation can be applicable? to real world situations.
And so that thread has run all the way through my career.
And I've been working on simulation at NVIDIA for autonomous vehicles for over five years now.
Yeah, you mentioned computational fluid dynamics and simulation has been such a big part of automotive development for decades.
When I was an automotive reporter, I would visit various development facilities for automakers and suppliers.
And there was always some kind of simulation rig in the facility.
But with autonomous vehicle simulation, how exactly is that used? today in comparison with these more traditional simulation setups.
Yeah, I think there's some very common threads.
One of the things that I think is really interesting is just early on in the CFD world with OEMs, there was kind of a little bit of a battle.
It always felt like between experimentalists and simulation engineers and there was always a discussion about what was real and how could you, which results were the ones that you should trust.
And I think that it's interesting because I think that that's mostly been resolved now in terms of how the industry operates.
And I think that if you go into any automotive company, any OEM that's building a car, I think that they would tell you that you were crazy to try and design a vehicle without using CFD.
And part of that is because the industry has gained a lot of experience.
And part of that is because the tools themselves have advanced significantly.
So because the computational power and the simulation methods have advanced. the ability of the simulation to kind of catch up to what was available in the real world, and in some cases far surpass it, have kind of been acknowledged.
I think that the industry, at least in the terms of aerodynamics, is definitely beyond kind of this is simulation useful question.
Actually, it's been interesting working in AV simulation.
I think that as an industry, we've moved here as well.
At the beginning, five years ago, there were a lot of questions about what kind of role should simulation play in the development of an autonomous vehicle, and is it useful, and if so, how and I think that we're beyond that now, which is a great place to be.
I think that in the industry, If you look around, I think that anybody who's developing an autonomous vehicle would say that you're crazy to try and do it without simulation.
And I think at one point early on, I think there was a discussion about, well, maybe we just do it with on-road testing or maybe we use some amount of simulation.
But Again, a lot of that progress has been made by two things.
One is experience in the industry, and the other is that our tools have improved.
And so if you look at the capabilities of what we have today versus what we have five years ago, there's still a lot of room for development and still really excited about the work we have left to do in simulation. but we've certainly come to a place I think where there's a lot of utility with what we have now.
What are some of the challenges with autonomous vehicle simulation?
So some of the basic challenges are always there, which are making sure that the simulation is providing results that are accurate and repeatable.
And that's really important. to have something that's trustworthy.
It doesn't need to be perfect to be useful, but it has to be repeatable and trustworthy.
Once you kind of move beyond that basic building block, then some of the challenges come to really how do you scale?
How do you build enough simulations or test enough things in simulation? to make sure that what you're actually getting is useful.
We expect that simulation provides us a lot of value in areas, what we call edge cases.
Those are the rare events that happen on the road.
This is where simulation really starts to play an important role because Those are things that on the road that we can't test.
A really good example is near misses or we're doing work right now with How do vehicles see car accidents on the side of the road, autonomous vehicles?
It's an interesting case. It's just not something that, you know, in our testing that we see very often.
And so when we're trying to teach an autonomous vehicle how to react and how to behave, If you don't have a lot of on-road events to really work from, then we can manufacture those in simulation.
But even with that case, the idea of how do we generate enough diversity and interesting, scenarios that we can actually get to a place where we're training something useful.
So scale is a big one. And then on the backside of that, It's easy to generate a lot of simulations, but then how do you as an autonomous vehicle development team, how do you consume all of that data?
One of the big challenges with simulation is it's really easy to generate so much data that you can drown in it.
And so figuring out a good way to not only generate scenarios and generate simulations, but then have an easy way to consume that data on the other side to make useful decisions is also a challenge.
That car crash on the side of the road scenario is interesting.
It hadn't occurred to me that that would be something challenging for the perception system.
For those who might be less familiar with this technology, the terms hardware in the loop and software in the loop come up frequently when talking about autonomous vehicle simulation.
Could you walk us through what those terms mean?
Yeah, sure. So both of those terms, software-in-the-loop and hardware-in-the-loop, refer to a closed loop simulation.
That's a good starting point. Closed-loop simulation just means that we're simulating a 3D world.
And then we also have autonomous vehicle software that's interacting with that 3D world.
So we're sending some sort of signal out of the simulator.
The AV software is receiving that signal and making some decisions. and it's sending control signals back to basically operate the vehicle in the simulator.
So closed-loop simulation, there's different modes and methods that we do closed-loop simulation and we simulate different things.
When we do software in the loop testing, basically just AV software, but we're not so concerned about what hardware it's actually running on.
So in the vehicle, there's often custom hardware that's been designed just for the autonomous vehicle.
And we don't necessarily need to run on that hardware.
We're just most concerned with what the algorithms are doing.
However, there comes a point in testing where the hardware starts to play a really important role.
There are questions oftentimes around timing or performance on the actual autonomous vehicle itself.
There's also other critical pieces of technology involved in terms of networking and safety and security. that are really important.
And so when it comes time to start running tests, where we want to be able to answer questions about not only does this software work, but does this software work on the hardware that it's intended to run on? then we need to start introducing hardware into the testing process and that's called hardware in the loop.
Hard run loop can be sometimes simple. It can be a single module or it can be very, very expansive. where there's all kinds of different vehicle computers that are introduced into the loop.
So you work specifically on the NVIDIA Drive Sim platform.
Could you talk about what exactly the platform is and how it came to be?
Yeah, for sure. So DriveSim is simulation tools that we've built for obviously developing autonomous vehicles.
We started several years ago building on top of a game engine called Unreal Engine.
And more recently, we've moved Drivesim over to a platform that NVIDIA builds called Omniverse.
And Omniverse has probably a whole other podcast, but there's lots of applications that NVIDIA uses Omniverse for.
Omniverse is essentially an engine for high-performance graphics applications.
So it makes a nice jumping off point for us to build Drive Sim.
DriveSim has a wide range of different types of simulation that it can do, but the thing that it really excels at is sensor simulation.
And so sensor simulation is essentially the process of building a virtual 3D world And then we call the process rendering, and that makes sense to a lot of people who are familiar with cameras, but we do the same thing for lidars and radars.
What we essentially do is we use that 3D world and then we generate sensor data or synthetic sensor data from that 3D world.
And the goal when we generate that data is to have that sensor data look and feel to the autonomous vehicle software, just like real sensor data, both in its appearance, but also in the way that it's, for instance, packaged or even transmitted over to the AV software.
And the goal is that AV software thinks that it's actually operating in a real world. thinks it's receiving real sensor data.
And the objective is that the neural networks that are running to do perception and the other algorithms inside of the AV stack think that it's all real data.
And so that's quite a challenging computational task.
But we've spent a lot of time in the software and with the hardware that we've built to be able to do that.
When you think about the level of detail in the real world, it can get mind boggling.
Now, how are you able to recreate this detail in simulation in a way that it does quote unquote, fool the neural networks.
The input to the sensor simulation is always the assets or the 3D content.
And traditionally, that's always been a challenge for us.
And it's been a challenge for anybody who tries to do sensor simulation.
One of the reasons is because the detail in the world is often infinite and trying to capture all of that is really hard.
And so we've tried to tackle this a couple of different ways.
The starting point or kind of the baseline that everybody works from is 3D artists.
Typically, if you build an environment or if you build an asset or anything else that you want to put in a prop, if you want to build a garbage can, you can have a 3D artist do that for you.
The problem with that is that it doesn't scale very well.
3D artists take time and cost money. And so if you want to get to an infinite level of detail.
There's only a finite amount of time and money available.
So we're working on ways to make that more efficient.
A couple of things that we're doing to make that more efficient are for 3D environments, We have tools to basically take an HD map, which we generate for our autonomous vehicles anyway, and be able to directly or automatically turn that into a real-world environment in simulation.
Some other things that we're doing, we're working very closely with industry partners to scale essentially the amount of assets that are being built.
We have some initiatives that we're working on to be able to develop what we call SIM-ready content.
There's no real standards out there in the industry, but we're working with partners.
So they all have a standard that they're working towards. with Drive Sim.
Makes it a lot easier for them to build content that is available right away and What that means is that a user of Drive Sim doesn't necessarily have to be an artist.
They can just go out and purchase something in a marketplace.
So if you need a garbage can, you can just go out and you can buy one.
Then we're doing a lot of research right now.
We have a really great research team at NVIDIA.
They do really amazing work. And some of their award-winning papers this past year, we're working on turning that into tools for us to generate assets better in Drive Sim.
So basically go out and drive and take real-world data and immediately turn that data into 3D assets like vehicles or props, or turn that into 3D environments.
Yeah, there was a there's a preview of that research at the last GTC.
I believe Nvidia CEO Jensen Wong showed a demo video Highly encourage you check it out to see this technology in action.
And on top of the detail and the environment, How do you make sure that the simulation scenarios encapsulate everything an autonomous vehicle might encounter from weather conditions to road conditions, to lighting, to traffic scenarios.
How are you sure that these simulations are comprehensive enough for testing and validation?
Yeah, that's a really great question and a really deep topic.
Actually, all of these, you could probably get a PhD in if you wanted to, I'm sure.
But validation is a really big challenge.
There's two types of validation and I'll kind of focus on the second one.
There's obviously validating the simulator. and making sure that the simulator is good.
We've already talked a little bit about some of those concepts in terms of validating sensor models and those sorts of things.
But then there's the question of how do I make sure I'm simulating the right thing to be able to test an autonomous vehicle and say that that autonomous vehicle is good.
We're kind of going back to our very first conversation about simulation just in automotive.
We're certainly not in the market to preach that simulation is the end all to getting an AV on the road.
Realistically, it's a smart combination between different types of simulation and on-road testing and testing with safety drivers and all of those different tasks.
Simulation really accept that in terms of being able to test some of those things that are harder to test on the road or to test them in a really repeatable way.
And so when we think about that and we think about, well, what is simulation good for?
It's helping us to discover what a lot of times in the industry we call the unknown unknowns.
So if you think about it, if you sat all of your really smart people down in a room together and you say, you know, let's think about everything that could go wrong with an autonomous vehicle and let's write a test for that. you would get only part of the way to your answer in terms of what you actually need to test.
So the other part of this really boils down to the methodology of those who are implementing the tests and NVIDIA has a separate team that we work with internally for developing autonomous vehicle technology. and we have a validation group and it's their job to work through this process.
And the process really revolves around being able to use additional tools to start evaluating scenarios and do some intelligent randomization around events. inside of the scenario to make sure that we start introducing unknowns into our tests.
And as we introduce unknowns into our tests and randomizations into our tests, we start to discover things that we weren't aware of before.
And so this is why scale is important. You need to do a lot of different tests to discover the things But there's ways that our teams work at this process so that We don't have to test everything and it's not just a completely random shotgun approach.
But we use directed randomizations and we start to search for places where we know that our AV stack struggles.
So we find the weak spots and we start to see how to exploit them.
And when we get there, then we can go back and we can fix those things.
Out of curiosity, is it possible to simulate the experience inside the car?
We've mostly been talking about the external experience. factors and the environment surrounding the car, but what about the interior experience?
Yeah, that's a really great point. So there's obviously the performance of the car on the road, but there's also the experience of the driver inside of the vehicle.
And there's two things that are important there.
One is just kind of a product question for anybody who's designing products an autonomous vehicle where it's actual product that they want to either sell to a user or they want that user to experience whether it's like an ADAS system or a RoboTaxi, the user experience is really important.
And so there's that question. But in certain circumstances, especially in the case of ADAS, user experience can also be a matter of safety.
So making sure that the user knows when it's safe or not safe to operate the vehicle, or if you have to do any sort of control handoff from the vehicle back to the user.
Making sure that those things are done properly is actually not just a Are you enjoying your ride?
But it's actually a question of safety. And so using simulation is a really great way to test what's called in the industry human factors.
So we can actually use Drive Sim to simulate the world outside of the vehicle, and then we can actually also simulate the inside of the vehicle.
When we simulate the inside of the vehicle using usually virtual reality or some other methods, we can basically put a user in a situation where they're actually experiencing what it's like to be inside of the vehicle.
When we do that, we can start doing some of those tests either just for overall user experience to make sure that they're happy or Some of the more critical tests for safety, for things like user takeover or collaborative steering or some of the other really important concepts.
One term that's been popping up recently in discussions about AB simulation has been synthetic data generation. which seems like a pretty descriptive term, but refers to a complex environment topic.
So could you walk us through what exactly synthetic data generation is and why it's important for simulation?
Yeah, sure. So when an autonomous vehicle system is built, one of the really important components of that is perception.
And the job of perception is essentially to take sensor input and use that sensor input to decide what is out there in the world.
It identifies cars and lanes and drivable space and a whole bunch of other different things.
The way that those networks most often in the industry today function is through AI and deep neural networks.
And deep neural networks need to be trained and the way that they're trained is using data.
So you basically have some sort of sensor data, something like a camera image, and then you have another piece of data that's associated with that, that's the ground truth.
And so the ground truth data is used together with the actual image to teach the neural network what it's seeing.
And so that's a pretty well-established process today, typically what they do for generating data to train neural networks.
It takes... most neural networks, hundreds of thousands to millions of data points to sufficiently train a neural network.
And so usually what's done today is companies will take a fleet of vehicles out on the road.
They'll go out and collect data. that data is brought back and data that's of interest will be usually sent to In the end, usually humans to go through and annotate the frames.
So as an example, you'd take a human annotator would take an image And they might draw a bounding box around anything that's a car inside of that image.
And then that pair, that set of data can be sent over to a neural network. help train it to get better at seeing things.
This process is good, but it has some challenges.
The challenges are typically that it's sometimes slow, inexpensive to do, but also that we have the same limitation in terms of edge cases.
So I mentioned earlier the example of emergency vehicles on the side of the road or things like accidents. the bulk of the frames that a data collection car will capture are usually of cars where it's following directly behind.
But it's very rare that you'll have an image, as an example, with a car lying on the side of the road.
And that happens in the real world, unfortunately, but it doesn't happen very often.
And so the question is, is how do we teach a perception system to see and recognize that there's an object there and then there's a situation that it needs to respond? respond to in a specific way.
And so this is where the idea of synthetic data generation comes in. situations that are rare but also oftentimes dangerous are difficult to get real data for, but we can actually augment our real datasets by generating these things synthetically.
So we can use the simulator to essentially position and place objects inside of the scene any way that we want.
And then we can generate that data and we can generate not only the image itself, But the other really nice thing about using synthetic data is we get the ground truth for free.
The really great thing about simulation is because we control it, we know where everything is and we know what everything is.
And so we get actually pixel perfect ground truth every single time.
There's no additional human errors or anything like that that gets introduced.
And so typically what we'll do is we'll use our standard set of real data And we'll mix in some synthetic data with that real data when we train our networks.
And we'll really focus our synthetic data efforts on on some places where our perception has some pun intended blind spots.
Well, we've gone from computational fluid dynamics to synthetic data generation and neural reconstruction engines.
What's next for the field of simulation?
I think a lot of it is really based around AI in the simulator itself. these are some of the research topics that we're working on right now that are really, really exciting, which is the ability to start mixing real-world data that's been collected buy our data collection cars and mix that with synthetic data in the simulator.
So basically taking real data and turning real data into a simulation.
There's a lot of really cool research that's being done.
There's actually several different research groups in academia and several other companies in the industry that are working on. tools like this and it's really exciting.
The thing that is most exciting about it is two things.
One is that it helps us move a lot faster in terms of the scale that we want for the diversity of our data.
And so we get all kinds of new data into our simulation a lot faster.
The other thing that's really exciting about it is that we start to reduce what we call the domain gap.
The domain gap is essentially the difference between what exists in the simulated world and what exists in the real world. and the data is never quite the same.
But as we start to use these AI techniques, we start to erase that domain gap because we're no longer simulating data, we're using real data.
And so there's a lot of really cool work that's being done that's really exciting.
And I think that in the future, we'll see, it'll be hard to tell the difference.
And it's always been that way in computer graphics, you know, for the last many, many years, every year things get better and we say, wow, this looks even more real.
It's harder and harder to tell the difference.
But pretty soon, we'll be using computer graphics generated from real data and when we hit that threshold, I think that we'll really have tackled the domain gap problem.
More widely, and you've been in this field for a decent amount of time now, what are your predictions for the next five to 10 years for autonomous vehicles?
Oh, that's a hard one. I mean, I think that we've perpetually been five years away from solving the problem, but I think that we're... we're getting really close.
And I think that the nice thing is that in the industry, the expectations are starting to align with the technology.
I think that we had a lot of really overinflated expectations for a long time. in terms of what would be achieved and when.
Certainly, I think that we'll achieve all of those things, but I think it'll just be in an extended timeframe.
But I think that the thing that will be really neat is new technologies that enable AV development to move faster.
A lot of the problems that we've talked about today in terms of how simulation scales and the types of testing that you can do and also just the tools that we give our developers to be effective, that we'll see a lot of improvement in those tools. and that will really make an exponential difference in their ability to be productive.
I think that's really what will be needed to move from some of the basic AV functions that are starting to appear in the market today. in getting to a place where we really get to true autonomy.
Well, thank you so much for joining us today, Matt.
We really appreciate you taking the time to talk through this really critical technology and what's on the horizon ahead.
So thank you so much for your time and looking forward to getting an update in the next year or so.
Great. Thanks for having me. Thank you.
