Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Before we get to today's episode, a quick thank you and a quick favor to ask.
It's been a little over a year since I took the host chair from the esteemed Michael Copeland, and it's been a blast.
So thank you to all of you, from me, from our producer, Brian, and from everyone else involved with the show. and at NVIDIA in general.
Thank you for listening. We'd love to get more of your feedback on the show.
So drop us an email. The address is AIpodcast at nvidia.com.
Send us your thoughts, your questions, your reactions. and even ideas for guests or future show topics.
That's AIpodcast at nvidia.com. Now for the favor.
If you like the show, please take a few minutes to do any or all of the following three things.
One, give us a five-star review on Apple Podcasts or wherever you get your podcasts.
Two, click subscribe. If you're not already subscribed, subscribe to the show.
You'll get all the episodes automatically.
And three, tell a friend about the AI podcast.
That's it. Thanks. Now let's have some fun.
We get episode ideas from all sorts of places.
Some we find in our daily work tracking the world of artificial intelligence.
Some come to us from individuals and organizations who want to come on the show and talk about the cool things they're doing with AI.
Some come from listeners who've recommended guests using, yeah, I'm going to plug it one more time, our email address, AIpodcast at nvidia.com.
But once in a while, some magic happens, and an episode idea comes from the same place we all turn to to answer life's most pressing questions. the search engine.
I have two young kids. They're into Legos.
I'm into Legos. My wife, yeah, she likes the Legos too.
So we have a lot of Legos at our house, many of which were built into sets and then disassembled and dumped into storage bins a long time ago.
So a few weeks ago, I was having a fit of temporary insanity or something and decided to try to do something about this big mess and started sorting one of these big bins of Legos.
Organized Legos are a little easier and a little more fun to build back into sets than just these giant messes, or so my sons tell me.
So about 10 minutes into my work, I had my, oh, there's got to be a better way moment.
And I hit the web. A few minutes after that, I was dashing off an email to producer Brian in a fever dream saying, Brian, I found this. guy.
He built a Lego sorting robot. It's using TensorFlow.
We got to get him on the show. Joining us today is Francisco Garcia, Paco Garcia III on Medium, which is important.
Francisco is the guy who I breathlessly told Brian about a few weeks ago.
He replied right away and he wanted to come on the show, which is awesome.
Francisco designed, built, and perhaps best of all, lovingly documented his work. designing a Lego sorter using TensorFlow on Raspberry.
It looks awesome. I want one. And Francisco is here to talk about the project.
Francisco, thank you so much for joining the NVIDIA AI podcast.
Well, thank you for the invitation. It's certainly extremely exciting to be a part of this. and see the work that I've done this year kind of translate into this great opportunity.
So thanks. Oh, our pleasure. It's very cool stuff.
It hits close to home for me and I would suspect a lot of the listeners out there.
So I read through the blog. It's a five-part series.
It's awesome. videos. It's got diagrams.
It gets into the nitty gritty of stuff, but it's very easy to read.
So kudos to you for that. And one of the cool things right away was was that you described stumbling upon the story of a Japanese cucumber farmer who saw Google's AlphaGo robot and decided to try deep learning to automate cucumber sorting at his parents' farm And then that inspired you.
So tell us about that. Tell us the story.
So that's where it started, right? So I have the fortune of working for a fantastic company and on my day job, I do technology.
But that's me. And when I come home, I'm typically writing my own code or playing with technology on my free time.
So I stumbled upon a couple of months ago with this Google Cloud Platform blog post about the Japanese cucumber farmer.
And it really captured my attention, right?
For me, it was, there's some places in the world where a car drives by itself and it can log millions of hours and It does it successfully.
But then you have like this small farmer with cucumbers assembling in a do-it-yourself manner. intelligent cucumber sorter.
And I spent myself like talking over and over.
I would use the example at work. And I think for me, that was, it's still something that intrigues me is like, okay, how, how is this going to work in the future?
Right. From self-driving cars. all the way to those kind of use cases in the rest of the world.
For people who don't know, there are a lot of types of cucumbers.
And there are a lot of things that go into deciding, is this cucumber, you know, first of all, is it suitable to sell?
Or is it a bad cucumber? But then all the different types and the things that differentiate.
So just to paint a picture, imagine... whatever it is for me, it's folding laundry.
One of those tasks you have to do that you wish could be automated and you'll kind of get the vibe, but, but sorry, continue.
Yes, exactly. And I think one of the things I learned is if the cucumber is straight, it's far more valuable to put it in the supermarket.
If it's shaped like a C, you don't want it.
It's low quality. So I really thought about this, and it stuck in my mind for many months.
And then I said, hey, you know what? I want to try it out.
I read several times a Google blog and says, well, they're using a Raspberry Pi and Arduino on the cloud.
It seems pretty straightforward. So I started giving it some thought of how could I put it together.
And I do have small kids as well. So I have an eight-year-old. a seven-year-old and a four-year-old.
And I'm a Lego fan myself as well. And we have Lego sets around the house.
So, and I said, hey, you know, I could really order the pieces.
I started playing and started sorting and seeing how many pieces I had of each color, each type.
But it really intrigued me that You can use Lego's pick a break system to order like the bricks you need.
Exactly right. So that was a plus. I said, hey, look, if I run out of pieces, I can always custom order.
But then as I was sorting them as well, I started identifying that around all our tons of Lego that we have, we have some I'm going to call them generic bricks in the mix, right?
So how do you, just to paint a picture, what do you mean by generic bricks? non-branded plastic breakers.
Oh, got it. Okay. Okay. They look the same as the Lego, but if you look really closely, they're different.
Yeah. And then I said, wow. I can order the pieces, I can design an experiment, and I jumped and I went online to the pick a break system with Lego, order a bunch of breaks, went to eBay and try to find those generic ones.
And over a couple of weeks, they're starting shipping in and I started building my sample set.
So that's how I ended up in the Lego sort of kind of motion.
Now, you said that your day job is technology.
Do you do AI? Are you versed in AI, deep learning, that kind of thing?
Yes. Okay. So not starting from ground zero here, but you have a background, have a little bit of, obviously, interest and some knowledge, and then thought... hey, let's try this with the Legos and the generic bricks.
That's correct. Okay. And my goal at that start, so I said, well, I'm going to do it myself.
I want to use Legos. But really what I was trying, from a personal perspective, I wanted to see how good could it be out of the box.
So I would say I'm going to build the automated machine to capture whatever sample size I need from images.
But I really want to see how good a neural network can be just out of the box.
And if you look at my blog post, I didn't I used very simple templates for the TensorFlow piece.
I used Inception v3, et cetera. I didn't do a lot in the software component, but that was my goal.
I says, how, how easy is it to get? Use the off the shelf stuff, see what it can do.
That's correct. Okay. Now I know, and folks listening out there know, there are a million and one kinds of Legos. let alone bringing in the generic brick variations.
So on your blog, you kind of laid out what your goals were for the project.
How many, let's start with how many types of pieces were you looking to sort through and classify?
So I started out with 20 types of pieces.
That's where I started. The first version of the machine ended up being able to sort 11 classes of bricks.
Okay. And this was due to like a design constraint that I had in terms of how many bins I could physically put You're doing this at home.
You have three kids. Can't take over the neighborhood with your Lego robot just yet.
That's correct. Did you start with... And I want to leap ahead because I'm fascinated by the sort of mechanical part of this, but... but we'll stay with the software.
I'm assuming you started with training your net you know, doing simulations, not getting into the hardware piece of this just yet?
So so what I did is before I started building anything, so so, you know, decided on Legos, decided on my success criteria, Googled.
I did the same thing as you did. I Googled a little bit and I ended up in this, again, with Google, this TensorFlow for Poets example.
And that's a lot of the base code that's running the the training and the inference of this machine.
Again, I wanted to do it very out of the box, very like let's see what's really available.
And from there, I got an idea and it really guided me like, hey, you're going to need like, if you're retraining a neural network, it'll take you about 200 images. per object.
So at that point I said, okay, then I need, I'm not going to be taking pictures individually.
I need an automated system. From that point on, I think I got stuck.
It was a good, I think maybe 80% of the time on this project was around the mechanical kind of piece of the project.
So getting into the mechanical piece, did you build it to take the images that you would then use for your dataset as well?
Or were you able to find Legos, without getting into the generic bricks, Legos are pretty standard.
Are there just vast image repositories out there you could use?
That was the core of the problem. How do I get the image set to train it?
And there's a million ways to do it. I chose going the mechanical route.
Some of my friends suggested and some of my coworkers said, hey, maybe you can use something like Blender and design some 3D models from their training.
Mm-hmm. Some other people around the net have done like, hey, you can use Google Images and kind of start tagging, et cetera.
Regardless of that, what I did found is that anything would require a lot of manual work.
So I then said, there's no way I'm going to be taking individual pictures or I'm not going to be taking images.
So this is a Lego sorter. And just to give the audience a visual, It fits on a dining table.
In fact, my wife told me, build it on the dining table so you're not away from the kids.
And I had them running around being part of it, but to hop in real quick, if you're listening and you're able to hit pause, go to Francisco's blog, there's a video, and you sped it up at 4x speed, I think, so you can watch the sorter in action. as you listen to him describe it.
Francisco, I'm sorry, go ahead. So the Lego sort of fits on a dining table.
It's made of two conveyor belts. On one end, it has a hopper. which is kind of like a bin where you place the Legos.
And it starts kind of separating it mechanically as it goes.
In the middle, it has one camera for the first version.
And at the other end, it has 12 bins. One of the bins is like a discard bin.
So it effectively can sort 11 types of Lego on the other end.
It runs with a Raspberry Pi. It has some motor and and several shields and some IR sensors, some LEDs to illuminate and give better quality of the images.
It mechanically separates pieces. So you start with a whole bunch.
So you just drop them at one end. It'll make them separate so it can take, it automatically takes a picture.
On the fly, it identifies if the picture has one piece.
Because sometimes if the separation doesn't work, you'll have two pieces of Lego in a single image.
So they run those very quickly and then start generating files.
Either for training runs, it'll generate an image repository like, hey, these are one by three bricks.
But when you're doing the real runs, it will run after taking the picture.
It will run an inference on the network and get the output of what the piece is and then sort.
So far, I think this machine in two to three weeks time frame I was able to go through around 10,000 pieces passed through the machine.
I took over 20,000 images during this time.
So on your blog post, on the one that's titled Lego Sorter Using TensorFlow on Raspberry, the first kind of overview post, you list out, you compare the Cucumber Farmer example from Google to the first run of your Lego sorter.
And you've got listed here, as you said, 11 classes, 11 separation bins, a training set of 2,300, time to capture the training set one week.
So is that coming from once you had kind of your strategy figured out and the machine built?
This is like the first real run. That's correct.
The first run I did was around identifying the break times.
What was your accuracy on this very first run?
So the end-to-end accuracy of the machine was close to 91%.
So to be exact, it was 90.8%. Whereas the cucumber farmer end-to-end was hitting 70%.
That's correct. So no disrespect because I love, my kids prefer Persian cucumbers.
I enjoy all types of cucumbers. Obviously, Google, Japan, a lot of love.
But that's phenomenal. 91% on your first run.
How'd you feel? Were you pleased? Were you just like, okay, this is good, but, you know, let's go for 95.
I was very surprised. Yeah, this is fantastic.
I think they're... It's two different experiments, right?
If you read carefully through the blog, it seems like the cucumber farmer used a different neural network, lower resolution images, et cetera, right?
They had far more many classes in terms of cucumbers, but nonetheless, I was impressed.
And this is out of the box, like this is literally running TensorFlow for Poets kind of script to train and deploy.
The training accuracy was really high, so it was 96.8%.
So I was losing some pieces in the mechanical piece, but overall, I was very impressed.
And I let it sit for a while. And I said, you know, how can I take it higher?
It seemed like a straight line. I said, like, I did the minimum, which is around 200 images per brick.
I said, well, I can increase the images, I can create features like flip the images and increase it.
But then I said, no, I want a bigger challenge.
So I want to see if this machine can help me identify real Lego bricks versus generic bricks.
And that was my second run. So before we get to the second run, if you just joined us, if you joined us partway through, we're talking to Francisco Garcia who built a Lego sorting machine that I'm, I want one.
Using TensorFlow running on Raspberry, did this on his dining room table, Wonderful documentation on his media or media on his medium blog, medium.com forward slash at.
Paco Garcia 3, P-A-C-O-G-A-R-C-I-A, the numeral three.
It's a five-part series. It's got a video clip.
Gets into schematics, all the details. Really, really cool stuff.
And as you were saying, you know, cucumbers, poetry, Legos...
I mean, that's the thing about where these tools are at right now, that you can access them, you can apply them to these different situations.
And so we're getting a little bit into the weeds for brickheads out there.
And I have some experience with generic Legos.
I've maybe bought my kids a couple of generic sets.
Maybe they kind of turn their noses up at me because they know a lot of times the generic ones don't stick together, whatever.
But there are subtle differences between the generic bricks and and the actual Lego brand bricks.
And so now you wanted a harder challenge and you're trying to sort between those two.
So how'd that go? It took me a couple of months to achieve it.
So the first thing I noticed, and this is more like, again, from the design phase, When I took a picture of the brick, it could be oriented in any direction, right?
And some perspectives on the bricks make it really difficult to identify if it's a Lego brand or a generic one.
So the first thing I did was I upgraded the camera I upgraded to four cameras.
So I took it from one camera, four cameras, and it takes pictures from multiple angles.
Okay. I run into a lot of issues, right?
When you're working with a Raspberry Pi, it's a tiny computer.
And I did really... I think one of the learnings here was like, you really got to check your hardware.
It turns out that for webcams, the other USB ports start limiting.
I think the device can only speak to seven USB devices at a time, et cetera.
It took me a while to figure it out, but got it up and running.
And this time around, I said, well, this is going to be harder, right?
If you look at the bricks, a Lego brick and a generic one, it's You can tell the difference, but it's very subtle.
And as you mentioned, you can take a look at the pictures and the difference yourself in the blog.
So I went out and I took... tons of pictures, right?
Like this time I have a set of around more than 20,000 pictures of real versus generic bricks.
And we're talking a couple of months time to assemble all that?
Exactly. So a couple of times to assemble it and roughly one week and a half to take the pictures automatically once I had the four cameras running.
Are you just having, do you have an automated system where you've just got a big bin of Legos and you hit the button and the machine starts going and they go through?
Or are you placing the Legos manually? Like how, you know, how much did you build here?
How sophisticated is it? It's pretty sophisticated.
So what I would do during the training phases, which is where the automation really, really helps, is that I would take, let's say, one by one breaks. right?
And I would dump them into one of the machine.
I would go around my day and it would take like 20 to 25 minutes to separate them, take an take the images, file the images, and then collect them at the other end.
So it required minimal input. What I did needed to do is I need to tell the machine, hey, this is What you're doing right now, it's a one by one break.
Right. Got it. And for those who don't know, one by one means it's the smallest it's got one of, they're called studs, right?
On the little round things on the top. Yes.
Yeah. So one stud on it, one, one by one square.
And I would tell the machine, well, this is a one-by-one brick.
And then I would tell the machine in another run, this is a one-by-one Lego.
Got it. And I would collect the images. Right.
It was very surprising. Again, when I got to the software piece, the neural network, I was still using the same kind of poet approach.
And I was using Inception v3 training a neural network on TensorFlow. which is if the audience is not familiar, the input size for the image is 299 by 299 pixels.
So in this day and age where we have kind of HD monitor, et cetera, 300 by 300 pixels, it seems really small.
Pretty low res, yeah. Well, it performed really well.
So over a series of runs and Working a lot on the feature definition, I was able to get it up to north of 80.85% accuracy.
Very out of the box. End to end. That's end to end, yes.
That's fantastic. Wow. And it was very, very, I was surprised.
One of the things I want to touch on real quick is the machinery itself You didn't, there's nowhere you can go to buy a dining room table, Lego sorting conveyor belt hookup.
You fabricated a bunch of these parts yourself, if I'm correct.
That's correct. So how did that happen?
And, you know, did you have the background in 3D?
Did you have a 3D printer? How did you do that?
So I had no background. I work primarily in software on that side of the house.
So the mechanical piece was completely new to me.
I did get advice from one of my friends.
And he says, hey, well, you might want to buy it.
3D printer. And I learned one thing, right?
Like the mechanical design by itself is a huge domain, right?
Yeah. I think it was key to have a 3D printer and the ability to go iterating little by little.
As I said before, I spent 80% of my time in that mechanical design.
I really was looking. I spent a ton of hours on YouTube, right?
Like, you'd be surprised, like, I had to fabricate my own conveyor belts.
I couldn't find something that was ready to buy that was really small and tabletop.
I don't know, maybe it's the industry, but small conveyor belts exist. but they're expensive, right?
They're made for industrial use. So something that was homemade, like I ended up like watching a lot of YouTube videos and, and, um, watching equipment manufacturers at the conveyor belts and see how they sort the pieces.
And it was really fun to put it together.
Very cool. Did the kids help or were you on your own?
Oh, definitely the kids helped. Yeah, nice.
They helped the most when I was doing the training runs. that's where they became more helpful.
Like they would fight over, to see who would refill the hopper so that it could capture more images.
I can imagine that happening in my house as well. you know, minus the system working because it's my house, not yours, but still.
So what next? Are you still working on it?
Have you pushed the Lego sorter as far as it can go?
Or do you have, you know, does this spark new ideas?
So after the second iteration, I decided that this project was complete.
So I've moved on. I've disassembled it. I'm storing it in label box, et cetera, so I could put it together in a flash if needed.
But number one, like one of the things that and maybe this is useful for the audience, but one is I took the time to document it and also share. the data set.
So if you look through the blogs, you'll find links to Kaggle data sets where you can, I want to say like there were so many people that helped me on this project. uh, with their content on the web that I said, Hey, if someone wants to try this out, I'm going to save them the mechanical headaches.
And there's a data set that you can use ready to go on Kaggle.
That's fantastic. Did you share your plans for the conveyor belt, the mechanical side of the house?
Not that you should. I'm just curious if somebody wanted to build their own.
How much of it can they just kind of follow the steps and how much of it is, you know, you have to figure some things out for yourself.
Well, my five part series of the blogs kind of illustrate the key concepts that you need to watch out for.
I didn't put out any schematics. But for example, one of the key problems that I solved was when you're taking a picture, and for example, you have a white piece, And the conveyor belt that I used was white.
Okay. So I couldn't identify the piece, right?
It would get lost into the background. So if you read my blog, you'll see that... that I show the audience where, hey, I was using a lead on the background.
And that way I could reliably detect the white pieces.
Clever. So all the kind of concepts and how do you separate the Legos mechanically?
Conceptually, it's there on the blocks? but no schematics yet.
Very cool. Are you sick and tired of Legos now?
A little bit, yes. It took me 200 hours over seven months.
But you asked me what I was doing today.
So I finished that project and... it has sparked a lot of interest and a lot of people like, you know, everyone's reading about AI and machine learning and there's millions of news going on every day.
And I got the opportunity to have someone reach out and say, hey, I want you to see what ideas can come if you see my factory. the opportunity to visit an injection molding factory.
And they have some robots that that they were not using.
And these were like 20 to 30 year old robots, right?
They were mechanically sound, but they were like out of pieces.
So it really picked my interest in saying, okay, well, how, what does it take to build a robot like this today?
I was very, very surprised. When you look at do it yourself stuff or tiny stuff.
There's a lot of open source. You can buy motors here and there.
Sure. You can see examples. But when you start getting in like kind of like an industrial workloads and the industrial kind of way of working, how factories, everything starts becoming proprietary, right?
So I'm very intrigued. Right now I'm working to see if I can replicate one of the machines that I saw. which is a robot arm that's used to pull injection molding pieces out of the machines.
Very cool. We'll watch your blog to see how that work goes.
Very, very cool. Any early insights or ideas around who is the problem that I need to solve?
Not yet. I think right now it's still a project.
It's still a hobby. But one of the things, if you ask me, like future looking, there's a concept that hasn't left my mind since I read the cucumber kind of blog from Google.
Yeah. So for me, I look at this space, the AI and ML space, And it feels like building a web page in the 90s.
So if you wanted to build a web page in the 90s, you probably needed like two or three engineers to work on it for a couple of months.
Today, it takes you three clicks and you have a web page.
I get the sense that that's exactly where we are from Ant.
AI perspective, right? It's easy to put things together, but it takes a huge amount of skills to do so.
And if I play it forward over the next 10 years, I think that's what we're going to see.
It's going to become extremely easy and it's things that took me 200 hours today. probably in 10 years is going to take one click.
Right. I like that analogy. That's well said.
Francisco Garcia, thank you for coming on the show.
Congratulations on the work. It's literally an inspiration to me.
I, you know, I'm super manual, but I've moved from Dixie cups to I bought one of those actual like 64 drawer sorting machines. sorting things.
So I'm taking baby steps. But the automated system is fantastic.
Again, I know I've said it 10 times in the past 20 minutes, but go read his blog, watch the videos, check it out.
Very, very cool. And as you mentioned, you've graciously published a couple of data sets so that people don't have to reinvent the wheel here and they can use your your images if they want to take this on or refine the system themselves.
So thank you for that. In addition, beyond your Medium site, anywhere people can go to see your work or anybody you want to shout out while you have the chance?
There were a lot of people, a lot of the on internet that helped me achieve this.
They're recognized in my main blog. And the reality is I got so much help and I learned so much from this experience that I also want to kind of like do the same and help other folks get on this journey.
So I'll keep posting as much as I can then as I keep documenting my work.
But feel free to reach out if you're excited about something.
And I'm more than willing to help. That's fantastic.
Francisco, thank you for your time. It's been a pleasure talking to you.
Thank you very much, Noah. It's been a pleasure and an honor to be here.
Thank you. Thank you.
