¶¶ Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Grab the goods and go.
Steve Gu wants to give every store, from mom and pop bodegas to supermarket chains, the ability to let customers saunter out of the door. without so much as a wave at a checker.
The benefits involve more than just convenience.
Stores will have a better idea of how their customers behave and get a real-time beat on their inventory.
To do that, Steve and his company, iFi, are relying on advanced sensor fusion, simulation, and deep learning.
Welcome to the podcast, Steve Gu, CEO and co-founder of iFi.
Thank you. I'm super excited about being here.
Thanks for joining us. So iFi is a Santa Clara-based company. right across the street, more or less, from NVIDIA's headquarters, which is cool.
And you are working on this... AI-powered, deep learning-powered, censored-powered experience so that somebody can make a transaction in retail, buy a good, but without having to stop and go through checkout.
Exactly. So how does it work? What can you tell us about it?
Yeah, so this is a new type of store experience and people just walk in store. grab stuff and walk out.
So there's no check-on lines, no waiting queues.
Everything is automatic. So behind the scene, there's very sophisticated cameras and sensor networks that keep track of your movement. understand who is grabbing what.
And when you walk out, you just walk out.
So there's no checkout. Okay, so let's break down the components a little bit then because we've got the shoppers, we've got the items in the store, and we've got paying for them.
Yes. So first things first, I walk into the store.
How does the store know who I am? It actually requires a cell phone app. in order to walk into the store.
So that cell phone app links to your credit card and the payment information.
But that's the only place where you need a cell phone.
Once you walk into the store, you can just put your phone away or put it in the pocket and the rest will be just tracked by the whole system.
Okay, so me and my identity being linked to my credit card, is that, am I geo-tracked?
Is it geo-fencing with this all going on?
It's all entirely based on computer vision and sensor fusion.
You don't need a cell phone after the check-in point.
Okay. And so I come into the store and I'm walking around and I
I used to live in Brooklyn. There was a bodega on my corner that I used to go to all the time.
It was a family thing. So I like that idea.
So I walk into the bodega and I pick up a bag of chips and a bottle of juice.
And I don't know what's a bigger object, maybe a big bag of dog food.
Yes. Right. Okay. How does the system know what I've picked up down to how much it costs and all of that?
Oh yeah, so there will be cameras everywhere in the store.
Not only they are keeping track of people, they are also keeping track of product items so they are watching the shelf 24 7 in real time so we will know precisely which item you grabbed so that's why we also say it's very good for inventory management.
And that's computer vision and deep learning.
It's not tracking a barcode on the bottom of the system.
No, it's the entire computer vision. it's basically similar to how the human eyes will behave.
Imagine that you walk into the store, your eyes can identify the product items, you understand which item you grab.
Basically, we're doing the same thing, just using Compute Vision.
And so what happens if somebody who's not enrolled with the program walks into the store?
In order to work in a store, you have to have a cell phone app.
But if not, then you have to go through the cashiers.
But if you do, then there will be a checkout-free lane, so you just go through that lane.
Right. Sure. Of course. So where are you in the development of the product?
Are there stores up and running? Do you have sort of a beta experience running?
Yeah. Yeah, so we actually have a fully functional lab store in our office.
It's about 725 square feet. We use that store for prototyping and testing.
The first pilot store we're launching, by the end of this year is going to be mind-boggling to many people because it's about 50 times bigger than Amazon Go.
So in that particular store, that store sells over 30,000 SKU product items.
So it's really different from the convenience store you just described.
It's more like a supermarket. And where is that store going to be?
It's on the east coast side of the state.
On the east coast of the state. Excellent.
So you mentioned Amazon Go. It's similar from what I haven't been to the store, but from what I've read about it, similar concept of not stopping at the checkout to pay.
And it kind of brings up slightly a broader question.
There are different approaches to every problem.
So these problems that you're solving, the retail experience and tracking people and behavior and inventory, that kind of thing, Is your solution unique?
Because there are ways that you're approaching these problems at iFi that might be a little bit different from what other competitors might be doing or just... other parts of the industry?
Yeah, I think our solution is quite unique and also it's very different from what Amazon Go is doing.
Mostly there are two sides. One is that we are really building a distributed computer vision system from ground up, which means that our solution ranges from a single camera plus sensors for a tiny mom and pop-up store and all the way up to hundreds and thousands of camera sensors for gigantic super centers and supermarkets.
So that itself is fascinating. It's meant to be scalable from the get-go.
Yeah, exactly. So that's why we think that this solution is going to be very interesting to third-party retailers because if you look at the Amazon Go, it's a tiny store, but also it's a customized built store, meaning that their solution is not that flexible. we really embed this flexibility into our solution from day zero.
So that's why our solution is quite unique.
Second is that because the solution is designed to work with existing stores, So we want to make sure that it has minimal impact on the existing source.
So the retrofitting process should become relatively easy.
So we actually devise the solution so that we optimize the camera placement, optimize the sensor placement, and also make it affordable.
So if you look at the Amazon Go solution, it might be super expensive.
But our solution, on the other hand, should be much, much cheaper and low cost so that it's affordable to the retailers.
And once it's installed, and obviously there's ramp-up time, but once it's installed, is the idea that it can be maintained by the proprietor themselves?
Yes, I mean, we can still provide tech support.
Of course, yeah. But the idea is that once deployed, It basically becomes infrastructure for the stores.
And the benefit is huge. I mean, we can enable real-time inventory management, we can collect individual purchase data, we can enable real-time analytics and charts about the shopping behavior.
These are all added value to the system.
Not to mention, we call it a reducer shrink because it's shoplifting things.
So how did the idea come to be? How did iFi start?
The company was founded in February 2016.
Initially, our focus was about edge computing.
By edge computing, what I mean was that we run very sophisticated neural nets on on low power devices, on IoT devices, cell phones, cameras.
So we actually did very well on that. We are pretty much one of the pioneers in being able to run real-time people identification and tracking and product recognition on mobile devices.
The next question naturally will be which vertical we went into.
We were very lucky that we talked to this particular retailer, which is one of the world's largest, and this project is super interesting to us.
And then delving deeper, we realized this market is huge.
And that really excites us. What were some of the challenges you faced or even some of the surprises you encountered as you worked to actually – tackle the problems of identifying the people and keeping track of the products and doing the contactless payment and everything else.
As you started to build out this idea that turned into the prototype lab store, what are some of the things you encountered?
First of all, this is a super challenging problem.
It's almost on par with self-driving cars.
Because the methodologies, the ideas, the techniques are very similar.
What's most challenging is that there are tons of corner cases there. especially that we are working with supermarkets, we want to minimize the impact on existing stores.
We want to make sure that the retrofitting is easy for the stores.
For example, that people can do all sorts of weird behaviors in e-shopping.
In supermarket, people can wear Pikachu costume, can do all sorts of... weird stuff.
So how can we make our algorithms so robust so that they can handle those situations?
That's a very challenging question. And also regarding the product items, most of the product items are fairly easy for I won't say easy, it's doable by computer vision, but there are also corner cases like closing.
It's super flexible, it's foldable, so it's very hard for computer vision to do that.
That's why we say it's a sensor fusion problem.
So ultimately, it's going to be 99% compute vision plus 1% sensors, including RFID tags and other sensors.
So to go back for a second to the idea of strange human behavior in stores, because when you said that, several things popped into my head at once.
Did you train your networks with data of people acting bizarrely in supermarkets?
Yeah, we do. That's actually one of the unique aspects of the company is that we really think really big about this domain and how we're going to solve these problems by going back to the first principle.
So the way we solve this problem is that we literally just digitize everything.
So we build a huge, gigantic store simulation. that digitize individual product items, the shelves, the fixture, the lighting, including the people.
So we build a very sophisticated avatar system that can simulate all sorts of different shopping behavior, like reaching to the shelves and exchanging items.
And also, we built the avatar so realistic that you can change the hairstyle, the facial expressions, the body pose.
You can wear all sorts of weird stuff, like a Pikachu costume.
We put all of these things into the simulation and making sure that the training neural net is actually robust against all those variants.
So that actually really empowers us to be able to deliver such a solution to big supermarkets and super centers.
Is shoplifting handled just the way it would be handled at a...
A regular store, I find a better word than regular than a store we currently have.
Are there things you had to account for for the idea of, when you're talking about the clothing I was imagining?
I don't know what this says about me, but I was imagining, you know, slipping a can of food underneath, you know, sort of a loose fitting shirt, that kind of a thing.
Yeah, totally. So that's actually another huge benefit of the system.
Once this whole infrastructure is in place, it's actually very hard to do shoplifting.
Because every time you grab some stuff from the shelf, you are being charged automatically.
Which means that this concept of shoplifting is going to be obliterated forever.
Stealing is buying. At least that's how we see this problem.
MARK MANDELBACHER- So once you're in the store, you're enrolled in the service, the store identifies who you are, that's connected to your credit card.
When you lift the object off the shelf, you're charged.
Yes. And when you put it back, then you get the refund.
You get the money back, right. So it doesn't matter if you're lifting it off the shelf to pay for it or lifting it off the shelf to steal it.
You've paid for it. That's it. It actually makes me realize that I stayed in a hotel recently where the You know, the mini bar with the snacks and everything had that sign.
There are sensors. Once you take it, you pay for it.
That's... Yeah, you can imagine that as a large-scale version of that big minibar.
Fascinating. We're talking to Steve Gu about, you know, the future of retail, basically.
We're talking about his company, iFi. They are using AI and deep learning, computer vision, something called sensor fusion, bringing this all together. to change the way that people buy things at stores.
And it's fascinating stuff and it's closer than you think.
Back up a little bit in time, before iFi, how did you get to this point?
What's your background in computer vision, deep learning?
Do you have a retail background at all? How'd you get here?
Well, I come from a very technical background.
So before this company, I was working at Google X, the so-called Moonshine factory.
I was in the Google Glass project, and there we did all sorts of weird stuff as well.
We'll put like eye sensors on the corner looking at the wink and the blink of the eyes.
And every time you wink at someone, you take a photo of someone.
So that's pretty cool. I also won a couple of innovation awards there.
For example, we put some magnetic ring piercing in the tongue, and it's your tongue movement to control the glass. screen so to control the Google Glass screen yes exactly Google Glass screen so I had a lot of fun and a very interesting experience there that never made it to market though right Yeah, unfortunately, in the end, the project wasn't super successful.
I know, and I was going to say that I think the backlash I remember in...
San Francisco and certain, you know, is banned from certain bars and everything.
You might've been able to get around that if you introduced the tongue pierce control, because it would hit your demographic.
Yeah, and we also did a lot of eye tracking and stuff.
So this is very interesting. And before Google Excel was working in Apple, the so-called human interface device group.
We did lots of prototypes regarding 3D touch and a false touch.
I actually helped to invent some of the features there.
Very cool. Yeah. So I got my PhD in computer vision from Duke University.
And actually, while I was doing my PhD, I was working at this company called Canista.
The company was a pioneer in doing 3D time-of-flight cameras.
That company was acquired by Microsoft in 2010.
It eventually became Microsoft Kinect 2.
So it was a very fun journey for me. So yeah, I don't have any experience in retail per se.
So I'm still learning about the retail domain.
It's actually more fun than I thought. How big is your team?
We have over 25 people. Most of them are research scientists and engineers in the field of deep learning and computer vision. but also from system software and the product development side.
So kind of stepping outside a little bit or stepping one foot outside of iFi a little bit and just kind of looking at the field generally, and particularly given your experiences working with you know, human computer interfaces and the different stuff at Apple and Microsoft and Google Glass, of course.
What excites you about the future of all this stuff?
Kind of the, you know, Near term or even looking further down the line, what do you think is coming down the pike that you're excited about?
Oh, yeah. So deep in my heart, I'm super optimistic about the future.
And I always promote these things called living in the future and look backwards.
So five years from now, you might just see that all those store operations become very automatic and there'll be like a self-driving cars deliver stuff to the store.
There were robots filling the inventory.
Maybe five years is too extreme, but maybe 10 years.
No, whatever it is. Yeah, so... I would really see that as inevitable because it makes everything better, more efficient. and that really frees the human labor and so that people can do something truly meaningful and more creative.
Now, when you roll out the store, not going to hold you to this, obviously, but the current plan, how many humans will be staffing the store, if any?
So, initially, I think you are going to see even more people in the store, more stuff in the store, just because This whole checkout-free solution at the store is not going to happen overnight.
Sure. we will probably see more people organizing a shelf, checking upon the shelf, making sure everything is working properly, and there will be even more cashiers and more staff members.
But over time, this will become more and more automatic to the point that eventually we don't need the cashiers anymore.
But of course, we still need people in the store.
Of course. I mean, they can... be engaged in those more emotionally engaged activities with customers, provide personalized shopping guidance, and so on and so forth.
No, that's a great point that it's not about replacing the people so much as it's about freeing them up to do More interesting things, creative things, or just things that you really need a human being to do.
Yeah, exactly. Excellent. If people want to find out more, and I don't know to what level of public or stealth you're in right now, but is there anywhere online people can go to learn more about?
Either iFi or your work personally? Yeah, so the website is there.
It's iFi.io. And we also have LinkedIn and AngelList accounts.
And feel free to send us email to hello at iFi.io. so that we usually respond within one day or two.
We welcome visionaries to join us. We welcome partnership.
And we look forward to working with more retailers to deploy these checkout resolutions in their pilot stores.
Excellent. Well, Steve Gu, thank you so much for joining us.
It's an awesome story and I'm super excited to keep track of it.
Look forward to your launch on the East Coast, you said, sometime next year?
It's actually by the end of this year. By the end of 2018, this year.
Oh, I actually forgot to mention that we're also opening a physical store in the Bay Area.
Oh, you are? We are. So we also expect to open that store by the end of this year.
And that store is going to open to the public.
We are going to invite retailers and consumers to just walk in and experience.
Oh, fantastic. That'll be great. Yes. All right.
Thanks for your time. Thank you so much.
Thank you. ¶¶
