Welcome to NVIDIA's AI Podcast and we are recording this segment from the floor of the 2017 GPU Technology Conference. a gathering of the AI faithful here in San Jose, California.
We've heard of robots working in warehouses, picking the right windshield or box of wiring harnesses from shelves.
But what about making sure those shelves are stocked with the right stuff in the right place at the right time?
Our guest on this segment of the podcast, Mark Junyoshi, thinks flying robots, you might know them as drones, are right for the job.
Mark is the CEO and founder of IFM, Intelligent Flying Machines, which it sounds like that's what you do.
Mark, welcome. Hey. Thank you very much for having me.
Okay, give us the vision here. Why drones?
We know there's robots in warehouses already doing all kinds of things.
What is this problem that you guys are looking to solve, and why are drones, or intelligent flying machines as you call them, the right solutions?
So we looked at the warehouse space in general, and what we realized is if you think about the average warehouse, Usually those are separated into two areas, the storage locations and the picking locations.
And recently, there has been a lot of work in the picking locations.
That's where you have the robots driving around on the ground delivering the parts to the workers, but the storage locations just haven't been touched very much because of their sheer size and scale.
And you're talking about warehouses. Let's be clear about this.
Not for me when I buy a pair of pants, but When there's a manufacturing situation going on, building cars, building, I don't know what else, prefab housing or something like that.
Today we're focused on manufacturing, but the warehouse structure in general is very similar across different industries.
But what we realized is that in the storage locations, the only way to really automate those places and to improve efficiency is to install what is called a fully automated warehouse.
And those cost upwards of $30, $40 million just to set up. where we see the opportunity to use these flying robots.
It's not that the flying robot is a solution, but it's a means to getting the data in those spaces.
And of course, there's a certain set of technical challenges that we had to solve to do this.
By having the flying robot, we can get the centimeter accurate data in those racks to figure out where everything is.
And we see our system really as a way to get these very manual places, these storage locations, and make them as efficient as the fully automated warehouses, sort of automating the humans.
Describe for us what the current state is, these kind of manually driven warehouses in the storage area.
I mean, you described racks, but what are we talking about? given a specific situation?
So you would think about, for example, seven, eight, nine levels of very big complete pallets that are stored in these racks.
You have thousands and thousands and thousands of these pallets stored along 50, 100 aisles.
And right now, people drive through with forklifts and they come and pick up and place the boxes in there.
Now, the problem is that because of the sheer size and scale of these warehouses, they might misplace the boxes every once in a while. and it's really hard to find them again.
So it's like Raiders of the Lost Ark, kind of, and it's still like that, right?
I guess, yeah, you could compare it to it.
And the important thing here to note though is these are not errors that people make, of course, by, these are errors that people make by accident just because it's so hard to reach those upper levels.
So you might just go to the wrong level and pick the wrong box or you might drop off the box at the wrong position.
And right now, In order to mitigate the risk of misplaced inventory, warehouses will buy more inventory than they need.
They will have a lot of people in the warehouse. manually counting boxes.
And that's a very inefficient process because they spend a lot of time just getting their pellets out of the rack, putting them on the ground, scanning them, putting them back.
And so what we can do is we sort of do that on the fly with the robots in a distributed manner. and each robot is completely autonomous in its operation and nobody's really controlling our system.
All the workers get is a guarantee that all the products they're supposed to pick are going to be at the right location.
And with that, they save time. and the pain of having to go and find and search for things.
So how is this an AI problem or a deep learning problem?
Like, what are you... training these robots to do.
Clearly there is image recognition involved, there's navigation, there's recognition or it has to sync up to a kind of a schedule or a platonic ideal of what the inventory should be at any given time.
So what are you teaching these things to do and how do you do it?
So the core technology that we've developed does two things very well.
One is the localization, which is just the traditional computer vision.
And then the other part is the perception system.
And the reason why we need this perception system, which is based on deep convolutional neural networks, is that all warehouses are slightly different. and the robot needs to be able to understand this is a box, this is a label, this is a rack.
And these are very high level concepts. For us humans, we look at a box and we understand this is a box.
But for the robot, it only has an array of pixels. some like matrix and it needs to understand that this like set of pixels corresponds to a label.
And once we find that label, we can position a robot so it actually reads the information from that box.
The thing for us is we're not trying to build a very general purpose solution.
And if you just take something off the shelf, you're going to have a network that's supposed to do very general purpose things, like recognize a cat, recognize a dog.
But we obviously have a very limited... very limited application field where we're only looking at certain parts in the warehouse.
So we were able to optimize parts of this network of the layers. and fine tune them to the specific application and the data that the robot is seeing.
Because for us, it's really important to be very fast.
Everything is happening on the fly on the robot in real time and very accurate.
We're trying to remove errors from the process.
So we can't really afford making mistakes in our detections.
So do these robots, are you saying that the robots can go into any warehouse and kind of get up to speed quickly?
But do they learn then the difference between these car parts and those electronics parts?
Is it all about a barcode and scanning the information and then they ingest that and know what they're looking for?
It's all about the labels that they already have on the boxes.
So we're sort of emulating the way that humans do the inventory count with a robotic system that's just 400 times faster than manual methods, and orders of magnitude more accurate, because robots are really good at counting.
Essentially, we just use the existing technology, which is the barcode that they have on the boxes, and we read that from the robot.
So how big are these robots? Can I call them a drone or do you not like that term?
I do not like the term drone because you can go and buy a drone at Best Buy for 50 bucks if you want to. we're building very, very specialized high-performance flying robots.
They have the NVIDIA Jetson platform on board. and they run our own computer vision and perception stack. and everything is very fine-tuned to this warehouse application.
So we built the robots from the ground up.
We designed them specifically for the needs that we have and remove all the bells and whistles that like a general purpose platform would give you.
And with that, we can get superior performance in terms of flight time.
We can fly in the dark. We can scan from long distances.
And everything is happening on the robot, which is pretty unique.
And there's no external infrastructure or markers or anything like that.
Wow. So, okay, flying robots, I get your point.
How does it work? You're saying they can fly at night, but how long are they out there?
How big are they? When you're saying they can go down to the centimeter, are they getting up close or can they just scan from a fair distance?
How does it work? So it takes us about 17 to 20 minutes to turn over an entire warehouse, at least the warehouses that we've seen.
And warehouses are like what size? Give me a ballpark.
So, I mean, really independent of the number of aisles, but you'll have 10,000 pallets in there.
Oh, wow. And the interesting thing for us is the way we design the system.
And it's really not just about the robot, but the way our system works in the warehouse in a distributed way.
And in doing so, we can essentially turn over the entire warehouse in that time.
And the robots themselves have a much longer flight time than that, which is key because we need to make certain guarantees about the robot position and the robot operation.
And for that we need a buffer, of course, on top of the mission time.
And do these robots fly when people are in the warehouse working as well, or do they do it all at night when everyone's gone, or does it really matter?
Depends on the customer. We have different customers with different needs for the inventory counting.
And our system is flexible in scheduling those checks.
So some customers might want continuous inventory checks. because their inventory is moving very, very fast.
Within several hours, they'll have turned over 60, 70% of their inventory.
And then others, they want to do it once a day or maybe once a week.
But the difference that you have here is manual methods if you want to turn over your entire warehouse. you will have to, for the same type of warehouse, same size, they would have to hire like 150, 200 temps and count for two days straight.
And we can do that in that short amount of time on a continuous basis without having to hire those 200 people.
How does the data get relayed back to the folks that need it?
Sorry, the flying robots are out there doing their thing and How does the information that they gather get sent back?
We integrate with the on-site inventory software.
So we essentially just emulate a manual inventory count, and we are able to then either update the information in the inventory software or specifically send the workers to the locations that need to be fixed or updated.
How did you get into this? Were you working in a warehouse one day and you're like, gosh, I wish I didn't have to climb to the top of this huge tower of pallets?
Where did this idea come from? I would say the genesis of this company is really a combination of my passion for flight.
When I was 15, I decided to build a flight simulator for 737, like a full-scale cockpit flight simulator.
From scratch. And then with 19, I started working in advanced robotics for automotive manufacturing on collaborative robots. and sort of with my experience in the robotics space and specifically an understanding of what it really means to bring robots into real world applications. and my passion for flight, I decided that there would be good synergy sort of between those two areas and a lot of potential. and everyone's focused on outdoor drones, but there's a lot of challenges both from the regulatory side. as well as the safety side which is extremely important to us that you can solve effectively in all outdoor scenarios and I saw an opportunity to bring drones indoors, and that's what we're doing.
Pardon the interruption, but we have a favour to ask.
Leave us a review on iTunes, Google Play Music or whatever your favorite podcast platform of choice is.
It helps more people find us. As always, thanks for listening.
Now, back to the good stuff. Are your robots at work today?
Do you have customers who are using them in warehouses?
Are you piloting it, no pun intended, with customers?
What stage are you guys at? We're currently in a great position.
We have a set of companies that we work with, and we're installing our robots right now.
How does this then fit into the future as you imagine it?
I mean, you talked about how this maybe is a period where automated warehouses are hugely expensive. yes if you build it from scratch maybe that's the way to go now but if you're retrofitting existing warehouses which is most of the world's warehouses.
This is another way of getting that automation and that efficiency.
But as you spin it forward, what is this, world look like and how do robots play a bigger and bigger part of it?
If I'm driving into a warehouse, you know, in the future, in the next five, 10 years, what what goes down i think we are we're focusing on this inventory counting application today because that's the way for us to commercialize this technology that we have.
But the bigger picture for us is to automate all data capture indoors because we see a need for your connected devices, your AR glasses, your smart your smart meters and all that to know where they are and what's around them and to know that before they like beforehand prior knowledge of the environment and With the technology that we're developing, we're enabling systems to know where they are and what they see.
And at a very basic level, that's the foundation of any autonomous system.
It needs to know where it is and what it's seeing, and then it can take an action.
And we see ourselves in the long term as a provider of that information. to all kinds of connected devices, specifically within the manufacturing space and industrial space.
What you describe are autonomous vehicles that, like you say, know where they are and what they're looking at and that applies to certainly driverless cars that we're thinking, you know, built working on today on, on the roads.
Um, But this idea of robots flying indoors is a fascinating one, and I suspect it's not going to be limited to manufacturing or to warehouses even.
Are there other applications that you can imagine where this kind of approach makes sense?
Definitely. I mean, being able to Being able to operate autonomously indoors is one of the toughest challenges in robotics.
And the moment you can solve that effectively for other applications as well, There's a ton of applications from security, surveillance, inspection, all across the board.
But One thing that people need to realize, and that's what we're working really hard on, the drone, as you see it everywhere, what we call the flying robot, is not always the best way to solve these problems, specifically because you have propellers that are rotating at thousands of RPM.
We see an opportunity in these high bay racks to use these flying robots to get this data very fast and effectively.
But with every other application, you always need to ask yourself the question, Of course it's possible, but is this the right and effective way of doing it?
And that's why we focus so heavily on our software as well, where we build out this very robust perception and localization pipeline. that happens to be running on these flying robots, but really it can be any device with a camera in the future.
Right, so the vehicle isn't necessarily the thing, it's the smarts, right?
Right. So it'd be like BB-8, you know, something rolling down a hallway or something.
Is indoors harder than outdoors in some ways?
Clearly, the margin of error is smaller indoors in a lot of cases. than outdoors.
But what are the different kinds of problems that you have indoors than you do outdoors?
So let's start with the one advantage that you have indoors, specifically with flying robots, and that's no regulation.
I thought you were going to say no wind, but no regulation is even more of an event.
But with that... On the challenges that you have, of course, the robot, there's no GPS.
So that's the first problem you need to solve.
You need to find a way to know where you are.
We do that using computer vision in our own SLAM pipeline.
But then on top of that, as you said, everything's sort of more confined. your precision has to be much higher.
So if you're flying in the rack, we need the centimeter level precision, and we can't afford any drift over long distances.
Whereas if you're outdoors and you're traveling kilometers, then that's a different set of challenges that you have.
And of course, if you're indoors, you also can make the platform very big.
So any robot needs to be safe. That's actually something that I wanted to bring up.
From my experience in the industrial robotics space, I saw sort of the change from taking what is like two thirds of all robots worldwide in manufacturing and automotive.
These caged big industrial robots and bringing them next to people.
So there was a collaborative robotic space.
And there was a set of challenges that were around, can you make guarantees about the robot's position, the impact points, and the contact points? and access to the workspace and all those things.
This will always apply to everything else operating autonomously indoors as well.
Ground-based mobile robots have an advantage because they're only in 2D, but then the moment you go into 3D, your workspace and your work environment becomes much, much bigger.
So making those guarantees becomes almost exponentially harder.
And it's not so much about can you do it, but can you do it safely?
I imagine you're building a company and it sounds like you're incredibly busy and so you may not have time to build flight simulators anymore.
What happened to that flight simulator, by the way?
That flight simulator is still in my parents' basement back home in Austria.
And every time you go home, are your parents like, okay, when are you going to get the 737 out of the basement?
I told him no, because now we've got our own office.
Ah, so you're going to bring it here. You guys are based in Chicago, and are you a product of a Chicago school?
How does that work? We're located in Chicago, right downtown, and we're out of Northwestern University.
I studied computer science. I just finished my undergrad degree.
And the team around me, we have three PhDs from Northwestern on the team and an undergrad.
And we're also hiring. So if anyone's listening to this who's interested in computer vision and robotics, And indoor robotics.
So you can't build more simulators. Your parents don't have more room in the basement maybe.
Are you doing anything else for fun? Apart from the occasional sports, when I have time, I usually go drone racing, which is a lot of fun.
And I imagine that your drone is not like any other drone.
What's the key to winning a drone race, by the way?
Being really, really damn fast. Thank you for that.
Well, I think I look forward to the day when there's things rolling and flying around.
And I love the fact that... This is an indoor versus outdoor problem, and you guys are taking it on.
Mark Tanyoshi, CEO and founder of IFM, Intelligent Flying Machines.
Thank you so much. Thank you.
