Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. A quick reminder before we get into today's episode, you can subscribe and you can leave us ratings, five stars please, on Apple Podcasts, Google Play.
SoundCloud, CastBox, and most anywhere else you get your podcasts.
And also if you have feedback, if you have questions, if you have ideas for topics or guests, We'd love to hear from you.
Drop us an email at AIpodcast at nvidia.com.
I don't know about you, but my house, well, it gets a little messy.
And I can't blame it all on my kids, my home office.
It's kind of a minefield. Imagine a robot who you could just simply say to, hey, will you tidy the place up?
And with a nod and an affirmation, the robot will get to work. clean things up, separate the junk from the not junk, and put things back in their natural places.
Sounds awesome, right? Well, it can happen.
Natural language processing, object detection, and robotic picking are cornerstones of the current wave of AI-powered applications transforming industries and, yes, promising to clean up our home offices.
Jun Hattori has his hands in all three as a software engineer for Preferred Networks, a Tokyo-based company with an American outpost in Berkeley, California.
June is here at GTC 2019 and he gave a talk discussing preferred networks robotic picking system, which lets users interact with it using everyday spoken language.
He also talked about the company's latest advances in object detection.
Jun Hattori, thanks for making the trip from Tokyo to Silicon Valley and for making time to join the AI podcast.
Yeah, thanks for the introduction. It's my great honor to be here and explain about our projects.
So I'll turn it over to you. Tell us what you're working on.
OK, so yeah, I myself is a software engineer who had the original background in NLP.
So I've been working for Apple as a software engineer for five years, then moved back to Tokyo to join Preferred Networks two years ago.
And since then, kind of working on the integration of the natural language processing technologies into robotics.
And after several months, I started to realize that robots cannot do as much as what we can actually instruct. advances in natural language processing or maybe spoken language processing are really rapid.
And now we can come up with a very kind of accurate system that can understand many of many of like humans are instructions but if the robots cannot do physical job as instructed, It doesn't mean anything.
So the easy example that I always think of is Siri.
Yeah. And, you know, imagine how it can – Siri can be great, but also sometimes it can be a little difficult.
Siri doesn't quite understand what we're asking, I'll say. send a message to my wife and a song will start playing, for instance.
But kind of imagine that, but extrapolated to the physical world.
So the robot needs to understand what you're asking it to do, which is the natural language processing.
But then it also needs to be able to do it.
Now, I alluded in the little intro to your robot that can tidy things up.
We were watching before the podcast started.
You guys have a pretty amazing video demo.
Tell us a little bit about that project.
Sure. So, yeah, we have recently announced a new project that is about a personal robot that can tidy up your room in a fully autonomous way.
And it comes with cutting edge computer vision and object detection technologies, as well as human-robot interaction technologies with spoken language interpretation and gesture recognition.
And by combining those two technologies, we have built a system that can work on the real-world task, automatically and also can communicate with people in an intuitive way.
So we're voice only here. It's not a video podcast.
Right. people can't watch the demo on our podcast, but can you kind of walk me through how it would work?
And when you say tidying up a room, Is the robot able to, does it learn which objects go where, which objects get get thrown away.
This is obviously much more sophisticated than a Roomba that just goes around and vacuums everything up.
So yeah, so let me explain how it works.
So first, the robots is kind of watch around the room and recognizes all the items in the room, and then it come up with a task list to move which object to where.
So the destination parties are actually pre-programmed for our demo system, but we want it to be learned automatically. automatically in the future.
And then robot has a task planning ready.
So it basically approaches one object one time, and try to pick it by considering the object shape and how it is placed, it kind of finds the right angle to grasp it.
And does it have, is it a pincher hand or articulated?
Yeah, it's a pinching kind of. Okay. It's called parallel gripper.
Right. Okay. So it's very simple. It's not as complex as like human hands.
Sure. But you know, you're actually building this.
I'm just dreaming about it. Yeah. That part is called grass planning.
Okay. And yeah, the robot tried to grasp an object stably and pick it up.
Then depending on the object type, the category of the item, it moves to the appropriate destination.
So all the clothing, towels, socks, are currently going to laundry bag, and trashes go to trash bins. and stationary or maybe pens go to the pen holder.
So it basically kind of classifies each item to the kind of various different places in the room.
And then how you mentioned the voice interaction and gesture interaction as well.
How does that work? Oh, yeah. So a robotic system can basically work in a fully autonomous way, but when any spoken command or a gesture command is given by the user, it can dynamically change the way it organizes the thing.
So while it is holding or maybe socks, it's kind of tries to put it in a laundry bag, but you can actually interrupt.
It's a robot. So like, oh, hey, so wait, just move it to the trash bin because it's already very bad.
Then the robot system can dynamically update its task planning result and move it to the TrushPay instead.
For the uninitiated, and I myself, getting to host a podcast for the past year and a half, have learned quite a bit about just how difficult You know, the object detection, classification and the robot picking process really is.
And so let alone folding natural language into this.
How hard is it? What are some of the obstacles you've kind of overcome?
And how has the evolution of AI and machine language, computer vision, helped you, enabled you to be able to get as far as you have?
There are many obstacles. we need to overcome.
But I think the first challenge was the very first object recognition part.
So we actually have kind of very good robot demo more than a decade ago by the Stanford group.
There is a robot called PR1. and it can actually do something similar or maybe more complex task than ours. so the robot can tidy up the mess in the room.
We have a video over 10 years ago, But at that time, the robot was controlled manually.
So robotics hardware was already ready to do that kind of complex task at that time.
But even after many years, we haven't been able to see some autonomous system The primary reason was there was no object detection system that can recognize such many items in kind of real world environments.
Recent advances in the computer vision community has finally enabled building such robust computer vision system for hundreds of objects.
And no matter how they are placed and regardless of the lighting conditions, it still works in a very robust manner.
So one trend that has emerged for me, and I think this is no secret to anybody who follows AI, GPU computing, computing in general, is that the past couple of years in particular, the advances in the hardware power that's available, and particularly GPUs and using GPUs for AI related tasks.
There's just been an explosion and that raw compute power availability has really driven a lot of innovations using AI related technologies and a lot of industries.
For the work that you're doing on this project, the Tidying Up Robot, and other things that Preferred's doing, have you seen the same thing as the hardware development over the past two, three, five years, really been a boon.
And then kind of looking forward. Are there obstacles that you're trying to get ahead of and you're waiting for the hardware to catch up?
Or is it more that you've kind of got the power you need and it's just a matter of figuring out how to use it?
Hmm. I think both the hardware and software development are important and the software side cannot do more than the hardware can So those kind of development are working in parallel.
And one of our missions was to make the full use of all the available GPU and the hardware technologies and really try to achieve state-of-the-art computer vision accuracy out of it.
So yeah, those are really important and software hardware has to be developed in parallel.
Another big obstacle we have is actually the data availability.
So even if we have the great computer vision model, as well as the hardware clusters, we still need to come up with a good way to collect data Especially if we are targeting domestic environments, we really need kind of huge amount of data for like people's personal lives.
We're speaking with Jun Hattori from Preferred Network.
A company based out of Tokyo, Japan, but June is here at GTC 2019 in Silicon Valley, and Preferred actually has an outpost up in the Bay Area in Berkeley, California.
California. Let's switch gears for a minute and talk a bit about your background.
You mentioned earlier that you came over to the States and you worked for Apple for a while. before returning to Tokyo.
How did you get into this field as a kid?
Were you interested in computers and data and robots and that kind of thing?
What led you to come over to Apple and what was that like?
So I did some PhD work in natural language processing and more particularly like syntactic on that kind of stuff.
I was looking for kind of good position in industry where I can apply. what I have learned and what I want to kind of work more on.
And many companies here in this area were very good options for me.
Sure. Did you do your work in Japan? Oh, yeah.
Back in Tokyo. And were you looking specifically to come to the US for a while or was it just it turned out it was a good opportunity?
Yeah, I was looking for a job in a couple of different countries.
I wanted to get out from Japan. Okay. And after getting interviewed by... some companies, I decided to join Apple.
And what did you work on when you were there?
So yeah, I was working on some of the NLP components on iOS and macOS.
Cool. Like kind of multilingual keyboard with predictive typing.
Right. Oh, wow. Okay. So at that time, I was like purely on NLP and engineering background, and I didn't know anything about robotics.
Right. And so what happened? How did you get into robotics?
After joining Preform Networks, I kind of discussed with the management what would be the kind of good target to apply such like natural language processing technologies It's probably known as one of the kind of difficult technologies to monetize.
Yeah, you have like... machine translation or that sort of things.
But yeah, we have like Google Translate.
So people. Yes. Yeah. Like. normal people don't pay for it.
And after some weeks of discussion, we kind of decided to first work on robots. like natural language as interface with the robots.
Right. And so we did some research work trying to operate robots using natural language.
And it did pretty well. What are some of the differences between applying NLP to something like a predictive text keyboard versus to humans being able to speak to robots and command them.
I mean, obviously there's the difference between typing and processing audio, Is that the big difference or are there other factors that people might not realize?
So the NLP for robots are Basically multimodal.
So it has to handle at least the speech and image at the same time.
So if we work purely on NLP, like for predictive typing, we basically handle text data only. we use various kinds of text data to come up with very good predictive model. but that is purely based on statistics from the text data.
For robots, If you're going to give a command, maybe pick up that white stuff.
Then the robot basically has to know what kind of items are there and how they are placed and what the white means or what borrow means.
So every word in your language is associated with some kind of visual appearances or the kind of state of the object.
So you have to do, uh the kind of mapping of the given word and the the given like image attributes or like yeah appearance yes that kind of stuff Right.
So it gets very, the levels of complexity seem like they would multiply right away.
Exactly. Exactly. Okay, so I'd be remiss not to ask, and it's partially selfish because, as I mentioned, my home office—
It's very rarely clean, let's put it that way.
Is a robot like this viable for mass production and you know sales is there a time in the future any idea how long until you know a consumer like me can order a tidying up robot Yeah, for the commercial product, we are really trying our best to make it available to customers as soon as possible.
So one thing we have publicly announced already was the – kind of aiming to release it to consumers within five years since the announcement last year.
That'll give me enough time to get my office in order.
So when the robot comes in, it'll learn this is what it looks like as opposed to learning, oh, it's supposed to be a mess.
All right. So that's good. So about four years, we hope.
What else was preferred working on and just kind of the industry in general?
Where are things headed? Sure. So prefer networks already basically specialized in deep learning type of technologies and always looking for kind of new industry field where such technologies can be applied right so so far we have a apply such technologies to the biohealth domain and also industrial robots and also the automobile industries.
But we have lots of new industries I cannot talk about at this point.
But yeah, we're always trying to expand our business.
And among that, horse and robots are one of our really kind of focused areas because everyone knows it has huge potential if someone can actually build something usable.
Yeah, in the coming years, I think they're going to be very kind of big competition among many companies or research groups.
So someone has to... prove first that there is a market and some kind of product can actually invade that market first, right?
June Hattori, thank you so much for joining the podcast.
If people want to find out more about what Preferred Networks is doing and also, you I think see some of your demos and your project pages online.
What is the website URL? We have the English web page of corporate at www.preferred-networks.jp.
And you can also find project pages from there.
Perfect. Well, again, thanks for making the time.
Thanks for making the trip out for GTC. Thanks for inviting me.
Hope you have a great rest of your show and good luck with all you're doing.
Thank you. Thank you.
