Welcome to Nvidia's AI podcast. We've discussed deep learning systems that are trained to play and master games like chess, Go, and even Texas Hold'em.
But training a deep neural net to play the racing game Mario Kart, what can you possibly learn from that?
Our guest, Kevin Hughes, has all the answers.
In his day job, he's a developer at Shopify, but he works on side projects on AI, and he studied AI and did a master's in computer vision at Queen's University in Canada.
Kevin, welcome. Hi. Let's just start with this.
Why Mario Kart? Were you an obsessive Mario Kart player?
Do you have posters of Bowser on the walls of your office?
What is it? You know, I've actually got a confession.
As a kid, I had Diddy Kong racing. I didn't actually get my hands on Mario Kart until I was in university.
So how did you pick Mario Kart and what did you do?
Why this project and why spend your time?
Getting the data, trading the data, and building the system to play this game.
Well, I've seen a couple other blog posts in the past about AI and video games.
It's not... specifically new idea, But I just kind of never felt that anyone really went deep enough and did it in kind of a way that other people would read those projects and say, hey, I could do that.
That's what I wanted to do with this project is I wanted people to read it and be, you know what, AI is not that scary.
I can do that. I could try that. And that's actually kind of related to why I picked Mario Kart, because it's just a very iconic game that is very popular.
Everyone knows Mario Kart and grew up with it. and I figured that if they could see an AI being trained on Mario Kart then maybe they would look a little bit deeper into what AI is and how it works.
Well, for the very few of us who don't know what Mario Kart is, it's a racing game where... gorillas and things with mushroom heads and turtles and you know other things race around on a you know how would you describe the track or the landscape or whatever it is I mean, it's a pretty standard racing video game.
There's a road and you try to stay on it and not hit things.
That's about the sum of it. Yeah. Well, so how do you go about starting this project?
And how do you go about kind of documenting how you did it?
Where did I start? So actually the first thing I started doing and, um, I had this finish and then kind of sat on this part of the project for about three years until I came back to it. is I set up my computer to run a Nintendo 64 emulator.
I wrote a Python program that would basically take screenshots of a section of my desktop, save the images, while simultaneously reading input from my Xbox 360 controller and saving that to a CSV file.
So then I could create this dataset of like, here's what my AI saw and here's what, you know, Kevin was doing while playing the game.
And then I could try to use some sort of AI technique to map the visual input to what the AI should do with it.
I see. So you're playing the game. The AI eventually is kind of watching you play the game or learning from your actions and reactions within the game.
Exactly. How many weekends and nights did that take?
Actually, ironically, this is what I was doing the night before I was interviewing for the job that i now have i would you know i got bored of studying like algorithms or whatever and just started doing this instead and stayed up till about 1am coding this thing that would read screenshots from my desktop and save them out.
I think I got mostly done that night in a couple hours.
I mean, it was pretty fast, it sounds like.
Yeah. AI code is kind of interesting because you end up, you don't necessarily have that much code around, but you kind of have problems that you know how to solve.
You kind of reach for these things that you know you've done and you can build stuff pretty quickly sometimes.
So the problem you were solving as it relates to Mario Kart was what?
What were you training your deep neural net to do?
Basically just to drive the course and follow the lines of the road to complete the laps.
And did that... translate necessarily to winning the game or did it just translate to finishing the course I never actually tried it in a racing situation.
I was really just focusing on the time trials and making sure that it could get through the laps.
My main goal was I wanted to train it on a subset of the tracks and then make sure that it could actually drive some of the other untrained stuff in a more economic way. more academic terms to make sure that it generalized.
Right, right. So it was actually, it learned how to drive as it were. and so whatever it was facing, it could handle in some sense.
Did it handle it? I mean, how did it work in the beginning and how did it progress as it sort of got better or learned more?
Well, I mean, there was more than a couple of roadblocks along the project if we want to get a little deeper into it.
So I came back to the project a couple of years later and chose a couple of new pieces of technology and started working towards getting it done. the first major roadblock that I faced was how was I going to get the output from my AI back into the emulator?
And it's possible that I'm just naive and didn't know the best way to do this, but...
Faking a joystick input to a computer is not actually trivial.
I thought it was going to be easy. I was like, oh, I'll just fake the joystick.
That'll be fine. But apparently Linux doesn't like that.
So I kind of had to like step away from my computer and I came back and ended up writing an entire custom input plugin to the emulator that would let me just feed data in from like a HTTP server kind of situation.
So that was roadblock number one. Then I had a pretty fun moment where I was hacking away on my input plugin and I wanted to make sure that I was correctly Programming inputs so I was like all right left joystick a hundred percent booted up Mario Kart and sure enough Mario drove in circles forever So I was like, all right, this is close Then I was like, all right, connected it all up to my AI.
My AI was sending input in. I was super stoked.
I trained the model and the convergence and everything looked good, set it up.
Mario just drove straight until he hit a wall.
I was like, oh crap, that's not looking so good.
It's crazy because what you're describing is you know, teaching a system to drive.
And you can imagine that when I say left, this is what left means.
When I say hard left, this is what it means.
And, you know. There's nuance that even if it makes sense kind of geometrically, it maybe doesn't make sense once it's kind of applied out there non-real world of Mario Kart.
Totally. Well, and that's kind of where the first thing that I found when I was going back to debug why Mario wasn't figuring out to turn with the road. is Mario Kart, the driving physics are really not very realistic.
And if you actually play the game, I recommend to do this, play Mario Kart, And when you are turning, you'll realize that you don't actually turn.
What you do is you course correct probably once every one second.
So you do a really sharp turn. then you drive straight, you do a really sharp turn again, and then you drive straight.
You don't really do like a smooth arc, So what that means is that the AI has plenty of data where it visually sees that it's in the middle of a curve, but it observes that the human player is hasn't actually isn't doing anything about that.
Then it just pretty quickly realizes it's like an optimization problem.
Well, it doesn't realize that it's not getting its goal.
That feedback path wasn't in the AI that I built.
Right, so Mario was driving like you were, essentially, or at least how it observed you were driving.
That's funny. Let me just ask you this.
Is that a little bit of a clue as to how to be a better Mario Kart driver? what you just described that, you know, you go straight and then you course correct, you turn hard and then you go straight.
Yeah. I mean, Maybe. I ended up training it with a very smooth curve.
I don't know if someone's, I mean, obviously if you get deeper in a Mario Kart, there's the whole power sliding mechanics that ends up being I think pretty important if you want to be like a Mario Kart Pro.
I never attempted to train my AI to power slide.
I think a few people on the Internet have though.
That's actually one of the cool things is other people have then downloaded the software and then trained their own AIs.
And I've gotten emails from people that indicate that they've successfully followed through all the steps to get it running, which is pretty cool.
So it works for not just you, but for other people.
Well, so look, you teach... Mario Kart to drive this subset of tracks and to do the time trial or whatever it is.
But what did you learn from that? And if we can sort of step back or expand this, how does this approach using video games Using training, let me say this again.
How does this approach, using video games as a source materialized data to train deep neural nets, How does that get more broadly applied out there in the world?
Well, so I think a couple of things. One, it gets more people interested in it and it lowers the entry barrier, right?
Because you can do this at home on a single You don't need a real self-driving car.
You don't need a robot kid or any of these other things.
You can start playing with virtual robots on your computer, and I think that's really cool because that's where you get this, everyone in the world can do this scaling effect.
I think that's specifically interesting.
Also, collecting data to build some of these AI systems is a big challenge.
And that's kind of what separates a lot of the big players in this field from the startups. where if you're Google, you have years worth of driving data, whereas if you're a startup, you don't.
But if you can spin up a platform couple clusters running Grand Theft the latest Grand Theft Auto or like a you know high quality video game if you can actually use that synthetic world to train real AIs, then that might be a really cost-effective way for new startups to acquire data and get into this field.
A lot of people are actually looking at this right now of semi-virtual synthetic environments for training AIs.
Yeah, I mean, you mentioned Grand Theft Auto and training people to drive and drive alone, I hope, but that it is sort of an open secret that that's happening right now and that companies in pursuing autonomous vehicles That's one of the way that they're getting their systems to learn how to drive is Grand Theft Auto.
Yeah, I mean it's got realistic physics.
We understand physics. It's got, you know, visually, the first thing your deep learning net is going to do is it's going to start deconstructing that image down into much more primitive things.
The first layer of the network is basically destroying the graphics in terms of actual information that's encoded, if that makes sense.
Well, wait, so you're saying that it's trying to get to the physics as opposed to, you know, get past all the beautiful graphics.
Is that what you're saying? Yeah, I mean, when your AI is seeing the road, whether it's looking at the very pixelated road of Mario Kart, the... almost not pixelated at all road of Grand Theft Auto or an actual road from the camera of a car. you want your AI to learn what a road looks like, and that concept is going to be much like it's not going to be related to how high fidelity the original road it saw was.
Right, right. It could be it's the sort of concept of a road and, you know, recognizing the right kind of. curves and in some cases stop signs or obstacles or whatever it is.
Yeah, exactly. And whether those were rendered by Mario Kart or by Grand Theft Auto or by some other simulation that is really out there in the world, It sort of doesn't matter.
Or does it? Like, do you think that, you know, how could you?
Well, let me put it this way. Would you get into a car driven? by your Mario Kart AI?
Oh, definitely not. So how do you make, well, why not?
And then how do you make that shift from hey, I'm great at video games, but out here in the real world, I need to think differently.
Well, I think there's a couple of things.
Obviously, companies that are doing this are logging thousands of successful hours, which definitely helps a lot.
They also have a lot more data that went into their model.
I pretty much played the bare minimum amount of Mario Kart to make this work, which was already a lot of Mario Kart.
So those two things make a big difference.
Then I think obviously for a real-world system, you need a whole second layer after your AI that sort of limit checks the outputs of whatever your AI would do.
I mean, this is getting way into implementation of a real self-driving car, which I'm definitely not qualified to talk about.
But the concept of training something on a game, as it were, and maybe that's game in quotation marks, but... that then shifts the learning into the real world, that seems pretty solid.
Are there other – there's driving. But are there other applications for that from the gaming world that you can see where –
You know, systems learn how to do something pretty well in a simulated world, and then they take that, you know, and tune that up for the real world.
Yeah, I mean, I think that's just a good strategy for data generation.
I mean, if you have a problem, I would definitely try to see if you can make it fit into that mold.
Right. So if it's you know, maybe it's you know, we talked about I mentioned at the beginning chess and go and poker.
Those are maybe too specific as far as games go, but are there even in those kind of game theory situations, are there other things that can be applied out there in the real world? by training systems to play those games really well?
Yeah, I mean, those ones are a little bit more specific because I think when you're talking about... like more of like an open ended game.
That's kind of where like, that's more of the domain that I understand with like visual learning, what you're seeing, what you think you should do something like chess and deep game tree that you need to look into the future and there's a couple of different concerns there when you're doing stuff.
But we've had flight simulators, so that makes sense because it's not cars, but it's planes.
But there's also... I don't know, sort of strategy games and other kinds of adventure games, I guess you could say, where you're... getting over terrain and you're solving problems.
And I guess, like you say, if you can find the problem that fits the problem you're trying to solve in the real world, then maybe a game is a good way to train it.
Totally. From what I understand, there's a pretty competitive scene for building Starcraft AIs.
I don't know what the application there is, but...
But people, I'm pretty sure, legitimately compete about who can make the best StarCraft AI.
What's the response been out there in the world?
You mentioned that other people have used StarCraft. your programs to build their own Mario Kart driving AI systems, Has it been extended further?
Have you seen other things in the Mario Kart realm or the things that you started, people took and ran with?
Yeah, so I mean, the response has been pretty phenomenal.
I mean, a lot of people have read it. It bounced around while it's still bouncing around on Twitter.
Lots of good feedback, lots of people reaching out that they want to get into this and try a project in this field if I have any advice for them, which is really cool.
I guess basically that was my original goal was to get people out there trying stuff.
There's been a handful of commits to the actual project on GitHub.
Especially Brian Zier has done quite a few nice contributions.
He cleaned up a couple of the things that I just did as quickly as I could to finish the project.
But his big contribution is he actually merged what I did with Mario Kart 64 and...
OpenAI's universe and gym environments, which is a whole wrapper for doing AI on video games.
I think they have all the Atari stuff, and then they do have the Grand Theft Auto, and then now they have Mario Kart 64, but more generally they have the ability to add additional N64 games to that and basically when you create a gym environment, that's when you're getting a bit more context. in the gym environment it's like okay here's how you start the race your goal is to do laps and it actually detects when you've made that goal and it opens up the um it opens it up to other AI things and you could do like reinforcement learning and just let this run on a machine someplace and let the machine give it the goal and let it try to get there. which is pretty exciting.
But yeah, I would really hope that somebody else picks up another N64 game and tries to do something soon.
Just to back up a little bit, what you're describing that Brian, this fellow Brian helped do, and thank you, Brian.
But it's an AI system that can play multiple games, right?
That can play as many as it's been trained on at this point, correct?
Yeah, exactly. He put a bunch of the plumbing around to make that easier to do.
What N64 game would you like somebody to tackle next?
You know, I was thinking about it a bit.
I think Star Fox would be a good one to try.
Star Fox. Now, Star Fox is outer space. What goes on in Star Fox?
I'm a little fuzzy on the details, but yeah, I seem to remember you're in outer space, you're flying a ship, you've got to get to the end of the level, and you've got to shoot down some things on the way.
There's a lot going on, and it's very visual, and there's a lot of 3D space, and I think it would be cool.
Someone should check it out. All right, Star Fox, get to work, people.
And for those people who want to get to work, What advice do you have?
How do you start to tackle these things?
And I think it's great because like you say, this is meant to pull people into the whole world of it.
What advice do you have for those people?
You just got to get out there and get your hands dirty.
The best thing about computers is they'll tell you when they're right, they'll tell you when you're wrong.
You can do it at home. It's great. Check out the project on GitHub.
It's called TensorCart. If you want to work around in there, I'm happily accepting pull requests and merging people's changes.
So if you want to even just start exploring there, that'd be awesome.
All right. Well, get in there, get to GitHub and get to work.
Kevin Hughes, thank you so much. All right.
Thank you. PostScript here. If you are listening to this podcast on iTunes, Google Play Music, or whatever your favorite podcast player is, do us a favor and leave us a review.
Thanks for listening.
