Hello and welcome to the NVIDIA AI podcast.
I'm your host, Noah Kravitz. Smarts are always in fashion, and our next guest has them in spades.
Costa Colbert has been exploring how brains, both artificial and real, work for the past 30 years.
Costa holds degrees in fields ranging from neuroscience to electrical engineering, and in 2007 he built perhaps the first large-scale GPU-based neural network simulator.
Now at Mad Street Den, his team is putting modern deep learning techniques to work for retailers in a wide variety of ways. including using generative adversarial networks to generate images of models wearing clothes.
Costa, welcome. Thank you so much for joining the podcast.
Thanks a lot. Happy to be here. So there's a lot to dive into here.
We were talking before we came on the air, and there are a million ways we could go with this, and we're going to get to them all in 20 minutes.
Well, let's start with... Let's start with right now.
For the layperson, what is generative... adversarial network and how, and more importantly, why are you using them to generate fashion images?
Yeah, generative – so generative network, just the general term, is something that can – they can generate something.
So rather than, let's say, doing a classification where you're trying to decide whether something is a dog or a cat, For a generative network, you're doing something like actually generating an image of the dog or the cat.
So the key there is for us right now is, sort of what's the range and the breadth of the images that we can generate, and particularly for us, the quality that we can get.
So how would you use this with a client?
One of the ways that we're applying the networks right now is trying to approach a problem that we've discovered in the retail business that has to do with fashion. it turns out that people like to see clothing demonstrated on other people, worn by other people, rather than just a disembodied garment, maybe lying on a bed or hanging in the air.
It's a lot more natural to see people wearing the clothes.
I like to see T-shirts on guys who are better looking than me so I can imagine, oh, I'm going to look good in that shirt.
Yeah, that's actually a really interesting point.
I think hopefully we can come right back to that because it is a very interesting point. interesting point.
So we were looking at this problem, thinking about the generative networks, and it occurred to me that if you could just take an image of the clothing and then just generate the person wearing the clothes that A lot of people would be very happy with that, right?
Now, it turns out it sounds kind of like the nemesis of the fashion model.
But it turns out that there's a lot of use cases where it sort of goes beyond just the issue of the of the photography, which turns out to be extremely expensive, and that's why the retailers have an issue with it.
But imagine a situation for, let's say, a consignment business. where they have a huge number of items that come in, but they only ever have one or two in stock of any particular thing, and they may never see the same thing again.
And so you can't imagine that you're going to be able to do photography It's expensive, but also just the time involved.
And somebody doesn't really want the added cost.
So if it's a consignment place and the clothes are relatively inexpensive and you don't want to have be adding some huge percentage in order to have paid for this extra cost, right?
So that would be the value proposition there.
The other part of it that's very interesting is that a lot of people, not everybody, but a lot of people like to see clothing on people that look somewhat like themselves.
So maybe their same build, their same skin color, something like that.
It's funny that you had mentioned before.
It turns out that that's not always true.
And there's what they call sort of aspirational photography.
Yeah, I have an aspirational ego, so it makes sense.
Yeah, so if you – so you may like to see what this – what this outfit might look like on somebody who you would like to imagine that that's what it looks like, right?
So it's not always a benefit to have something that looks too much like the person. but the opportunity is there.
And then the other part of it has to do with there are more and more shops online that are coming around that are being able to make clothes on demand.
So you may be able to pick a Let's say some style of a garment, but then you can tell them I'd like this particular collar, this particular length, this particular sleeve length, that sort of thing.
And then they can make it for you on demand.
So you can imagine in a case like that, there's no way that they can take pictures of every possible combination for everything.
And so this would be good for those sorts of people. to be able to just generate it on the fly based on the specification that the person asked.
Right. Now, if in any way, how is your approach to generating the images different than what other people might be doing?
Yeah, that's interesting. So one of the biggest differences here is that if you're trying to do something for an academic paper, You're trying to show a new technique, show a new way of doing something.
You can show a few examples and you get your paper published and you move on.
What we're shooting for is to actually have a viable product.
And so to have a viable product, you need to have the majority of what you generate actually available. look really good.
Right. So we have a little bit different approach in just in the way that where maybe maybe you could say the level of care over the long term. of how we're putting these models together to be able to reach that sort of goal, to have a very, very high percentage, high quality sort of output.
The other issue is that Traditionally, and it's just changing just in the last few months, but traditionally in these generative networks, most academic papers have generated pictures that are literally only... say 64 by 64 pixels or 128 by 120 pixels.
And if you think about your digital camera, that's just like one little tiny, maybe it's the person's face, right?
So we're working very hard on a number of different ways to be able to have much larger images that still have good quality.
And thankfully, just in the last few months, some other people have come out.
NVIDIA actually has come out with some papers where there's some very good progress in those areas that we can leverage a little bit.
So tell us a little bit about Mad Street Den.
I know you're based in California, but the whole company is not.
Fashion retail industry photos are not the only thing you guys are up to.
So the company came about, I had been...
Living in Houston, I guess about 15 years, I was a postdoctoral fellow and then a professor there.
And towards the end of what ended about – it's been about 10 years since then –
Moved out to the Bay Area from Houston. Worked in a small company for a couple of years.
And I met one of our other co-founders, Anand Chandrasekaran, and he and I really hit it off because we both had electrical engineering backgrounds.
Turns out that he also had gone and gotten a PhD in neuroscience.
And so we were both fairly technical people that were very interested for a long time in how the brain works.
And so when the opportunity came up, I was able to give him a contracting job.
And so he worked for me for a while. And once a week he would come over and we would spend the morning working on what we supposed to be doing.
And then we would spend the rest of the afternoon talking through some theories that we had about how certain things in the brain would work.
But also, what could we do to make a business out of this?
Because we had both been academics and we'd both sort of spent enough time working very hard at things that very very few people actually were interested in.
And it's never bad to think, OK, maybe we can make some money for this.
So we start talking about it and did quite a lot of work.
And then, At some point, his wife, Ashwini Ashokan, who had been at Intel for 10 years and a product manager at Intel for 10 years, basically said to us, well, let's either do it or stop talking about it.
And so she took on the role of CEO And Anand took on the job of being the CTO and put together a very good team.
Now, at the time where they were putting it together, they decided to go back to India.
And so even though we have a very close relationship, we're halfway around the world, But the advantage that we have is that we have an extremely strong team in India.
But we have them there actually guiding the way.
So we don't have to really run the company by remote control, even though many of the people are over there.
And then I have a smaller part of research team over here.
Cool. And what else, if you can talk about it, what else are you working on?
What we decided to do, we got into the retail and fashion.
It was mostly, we'd done a few different experiments and we were trying to decide... what would be a good area to focus our efforts and make some specific products.
Then, having been in that particular space for a while, we started to realize that a few things about how to run the business, but the main issue was that we noticed that for e-commerce companies, that there were certain pain points that they all seemed to have.
And it had to do with having consistency in their catalogs and being able to have the best presentation of what their wares were on their websites, having being able to make good recommendations so that people can get through a large catalog very efficiently.
Because if people get bored on a website, they just move on, right?
So we were focused on that sort of thing.
And so we identified quite a number of different products in that whole sequence. instead of just being a company that's sort of had, you know, one trick pony sort of thing, we've been able to generate a whole you know, a whole slew of them.
And so the kind of things starting on sort of just tell you like the two ends, the one end has to do with being able to for the catalog consistency is to be able to help clean up catalogs in terms of finding duplicates,
And then recently, something that's been very important for a number of retailers is... to be able to have the metadata that goes along with with a particular product.
So let's say a shirt comes in and you might have a metadata record that says what color it is and what style it is and it's buttoned down and that sort of thing.
Right. And interestingly enough, there are a lot of mistakes in this data.
And it's compounded extremely... When you have a company that's an aggregator and might have a thousand vendors sending them something.
So even if there are no mistakes, there's a problem with the language is different for each vendor.
And you cannot get people to use exactly the same language.
That's not going to happen. And so companies have this problem where even something like the description of a color might be completely inconsistent, right?
I just had flashbacks to catalogs of all the names for Periwinkle and Sea Mist and what have you.
I'm making things up. There you go. So our approach has been to apply computer vision techniques to be able to look, just look at the pictures.
Right. do the whole thing from the pictures, look at it, be able to identify, let's say that it's that it's a women's long dress, three quarter sleeves, floral pattern. some kind of buttons, some kind of neckline collar, all these sorts of things, and be able to describe in a consistent way what all the products look like.
So we think that's a particularly good use for this type of stuff.
Then it goes on further to things like recommendation systems and personalization, where We can rather than let's say let's say you pick some garment and at the bottom there's a little carousel that shows you five other choices you might be interested in.
But rather than it being some kind of statistical thing about what the last 10,000 people might have liked, which is collaborative filtering, we're able to start making predictions based on which things you've been clicking on about what type of style you like and what type of colors and whether in this particular time you're looking for the pattern versus the color. and that type of thing.
And those sorts of things are even much more interesting when you have somebody who has an account at the particular store, and then you have information for multi-session.
Yeah, you have their history, right? And so when somebody comes back in after a few times and you know something about what they like, then you don't have to start showing them all kinds of nonsense that they don't want to see.
Right, right, right. We're talking today with Costa Colbert.
He's the Senior Vice President and Chief Scientist at Mad Street Den.
And we've been talking about Mad Street Den's Use of deep learning artificial intelligence to generate images and make images. the retail experience just better overall for retailers and shoppers alike, particularly in the fashion industry.
But Costa, we touched on it and you hinted at it a little bit in talking.
You've got a pretty deep... varied background in academics and background as a neuroscientist, electrical engineer, computer scientist.
So let's step back a little bit and And I mean, the only way I can think to ask the question is back up from fashion.
How'd you get here? What have you seen? I know you've got.
Got a lot of, forgive the pun, but a lot of deep knowledge and deep learning.
So, you know, let's walk through it. Pick a time in your past and let's move forward.
Yeah, yeah. See how quickly we can. move through a lot of years.
So I started the first time I got interested in artificial intelligence was 1980.
Okay. And I had a summer job that was very boring.
And I pulled a couple of books out of the bookshelf that happened to be in the office I was sitting in.
And one of them was on the programming language list. which has been sort of the basic language of artificial intelligence for symbolic artificial intelligence for decades. and started reading that book, thought it was very interesting, and then realized that the whole second half of the book had to do with artificial intelligence, how to write programs that play games, how to write programs that can decide which blocks to move around.
Things that are still coming up in the talks today, the blocks moving around.
Going forward, I was interested in going to medical school.
So I was sort of interested in the biological end of things and very, very quickly realized that the basis of learning and memory in order to understand that that we really needed to understand how neurons work and how synapses work right and in deep learning that's the whole game you have layers of neurons and the neurons are connected with what's an approximation of synapses.
And based on how strong the synaptic connections are between the neurons, that's how you store information.
So it became very clear to me very, very early on that if we were going to understand anything about how the brain works, that we needed to understand that, just like you need to know about transistors.
Sure. before you could really start to do anything else.
So I put my eggs pretty much in that basket and went down that path. and spent quite a lot of time trying to understand how information is stored and how neurons process information.
So where that leaves me now, which is kind of an interesting thing, is that most of that knowledge at the moment in 2018 is fairly irrelevant to the present effort.
So looking forward, I'm expecting I'm going to make a little prognostication here.
I'm expecting that in the future at some point that The realization will come in that there's a lot to learn from what nature has figured out. which although that has been our premise for a very long time, although people right now tend to think that that's not so true.
I think that that will happen. I think we'll find that there's still many things that biological brains do much, much better than computers.
Has that been your experience recently, that there's a sizable number of folks who think, oh, we've learned all we need to know from nature?
Yeah, you have to be careful how you say that to not to be snarky about it and not to put people's efforts down.
I think what I can see is that there was a time until not too long ago when people would often start machine learning and computer vision talks, discussions, with sort of a nod to the biology and always say well we're not as this isn't the brain right but we can do good things anyway okay I think what's happened now is we're seeing more an attitude of we're super, we can do superhuman.
Now, some of that ends up being the marketing side of things.
But there are many, many tasks. where I think that we didn't appreciate before that really do sort of fall into what's essentially the standard machine learning And I take deep learning to be part of machine learning, even though the really hardcore machine learning people don't necessarily think that's true, oddly enough.
So I think now it's a little bit less likely, and I think there's more.
Or, for example, I think also now there's more people, you know, there's back propagation, which is the sort of basic algorithm that people use to decide how to train weights and how to train networks. and a very, very powerful algorithm, which for years and years and years, everybody assumed that you could that there was no way the brain was doing that.
Okay, well now, You know, even Jeff Hinton, who needs no introduction, has talked about how most of the arguments about how the brain couldn't be doing it are probably wrong, right?
But personally, I still think that the brain really is not doing back propagation.
But I can guarantee that over the next few years that those waters will be muddy.
There'll be people both on the neuroscience side and on the machine learning side Maybe trying to establish that, maybe trying to in some way justify that the way that we're doing things now is somehow an optimal way.
This feels like a little bit of an obvious or naive question as I'm saying it, but how much kind of over the history of your career and then to this moment and then going forward, Do you see sort of the biological people, the neuroscientists, the folks who study the human body, human brain?
Yeah. and the computer people, so to speak, right?
The data scientists, the programmers kind of coming together And is that kind of a more recent phenomenon?
Is that something that you think is going to be part and parcel going forward?
You're not the only one, but a little bit unique in having this medical background and now being involved with...
Yeah, certainly not unique. And certainly in terms of the medical imaging, there's so much deep learning going into medical imaging now that any kind of statement like that is really- in the past now.
Interestingly enough, there's a long history in the neuroscience field where people you have to remember that the majority of neuroscience and neurosciences is a biological field.
And the sort of people that tend to be interested in biological questions are usually a different crowd than the people that are interested in, say, physics. less mathematical, more interested in, I don't know, physical things.
That doesn't work now. People that are just less interested in mathematics and explaining things by sets of equations and more just getting their hands dirty and seeing what's happening there.
And so there's kind of a history looking at people in neuroscience that the people who are doing what's called systems neuroscience, where they would, let's say, be doing an experiment maybe with a person and showing them, doing psychophysics experiments, showing them something on a screen and then measuring something about how they react or how they learn something or or teaching an animal to do a particular task and studying those sorts of things.
Those people would think about computational questions, but The people who in the sort of neck of the woods that I was in, which to me was truly the nuts and bolts of the whole thing, the synapses and the neurons, was completely dominated, still is dominated, by people who are, at the end of the day, really much happier that somebody else worries about the theory of how this might work and just try to get the exact parameters out.
So there's always been a disconnect. and there's in the neuroscience community.
So the computational neuroscience was always pushed to the fringe a certain amount.
And I think now maybe that will start to change because there'll be a lot of younger people coming up who see the value of both of those things.
So I think that we have some exciting times to come.
So kind of given where you've come from and your sort of multidisciplinary background and to where you're at now, I'm sure you rub shoulders with lots of young data scientists and engineers who are maybe not just starting out in the field, but these days, two or three years of experience. sometimes counts for 10 years in the past, so to speak.
What kind of advice would you give folks as far as both doing what they're doing now, but keeping an eye towards the future and kind of being I don't know, prepared and opening up their minds for the possibilities of what might be coming down the pike.
Yeah, so for somebody coming up, I think the key thing is just to make sure that you have as broad an education as you can possibly get, whether that's... as an undergraduate or in graduate school.
There's a tendency in graduate school for people to focus you on a particular project in a particular area.
And you have to make sure that you have as broad an education as you have, because I can guarantee you that things will change. and change significantly.
And if the whole world and the emphasis changes, then you have to be ready for it.
Otherwise, you'll find yourself very unhappy in a job that you thought was going to be wonderful.
Let me make one point about that. I've been coming to this meeting for 10 years now.
First time was in 08. And they were just starting to talk about CUDA and just starting to talk about putting GPUs to work for all sorts of things.
I was lucky to have been added already for a year with a very, very early like 0.2 version of CUDA.
But what I'm saying is that You look at this meeting over the course of five years and over the course of 10 years, And it is so entirely different.
The tone is so entirely different. The focus, the jobs that are available, everything. has changed so much that you need to be a little bit careful that over the next five, over the next 10, over the next 15 years,
It would be very, very surprising if most of the knowledge that you need to know then doesn't really exist now.
And you need to be in a position to be able to make use of that.
And that takes a broad education. JOSH SHARFSTEIN.
And so that makes my next question entirely unfair, but I'm going to throw it at you anyway.
Where is this all headed? Where do you think five years from now are we gonna have a significantly more accurate virtual human brain?
Are we going to- Oh, yeah, yeah. So I think- You can take that from any angle you want.
Yeah, so I think I'm a lot more positive now than I was even in the last couple of years.
I think that we've hit a point in this industry where there are so many good things coming out of it and so many –
Very, very talented and smart people that have gotten into this and have started to produce a lot of things.
So I think the industry is really here to stay.
But it's a whole area of problems that's incredibly broad that we can make that using what we have, whatever name you stick on there, you know, is incredibly useful and allows us to do things that even just a couple of years ago we really would not have expected.
So very optimistic about the future of this.
It's a good place to be in. All right. Kosta Colbert, thank you so much for taking the time to talk with us today.
If folks want to find out more about... You know, either your, not to say it again, but your long-storied, broad-ranging career or the work you're doing right now with Mad Street Den.
Where can they go online? You can go to the mattstreetden.com website, or actually it's easier to get there by view.ai, which is V-U-E dot A-I.
And what is Vue? Vue is actually just our retail vertical within the company.
Very cool. Well, thanks again for taking the time and best of luck with wherever life and deep learning takes you next.
Thank you very much. Thank you.
