Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. During her company's recent Q3 2019 earnings call, Stitch Fix CEO Katrina Lake described one of her company's measures of success.
One measure we look at is the number of clients who keep at least one item in their fix, she said, and who tell us that they're looking forward to their next fix.
So what's a fix? A fix is a box full of curated clothing and accessories sent to clients, Stitch Fix customers, once a month.
How does the company ensure its clients look forward to their next fix?
Through a little bit of art and a little bit of science.
Data science. Brad Klingenberg is Chief Algorithms Officer at San Francisco, California-based Stitch Fix, where he manages the data science and algorithms teams that help optimize the Stitch Fix client experience, the management of inventory, and the selection of items for clients.
Brad's here to talk about the marriage of art and science fashion and algorithms, if you will.
That's helped Stitch Fix become a successful publicly traded company and is shaping the future of AI in retail.
Brad, thanks so much for joining the AI podcast.
Thank you, Doug. Great to join. I'm actually a little bit familiar with Stitch Fix.
My wife is a client, full disclosure. But why don't we start?
Why don't you tell the audience what Stitch Fix is?
I'm going to trip over Stitch Fix this whole podcast.
I apologize in advance. I'm going to do my best.
Why don't we start by having you tell the audience a little bit about what Stitch Fix is and what your role is and how algorithms and AI figure into the business?
Absolutely. So Stitch Fix is a online personal styling service for men, women, and kids.
This service is really one of personalization where a client can request what we call a fixer or a shipment And based on preferences that the client shares, we'll choose some things to send to the client that we think they're going to love. and really act as their personal stylist.
And so the key distinction from traditional retail and even most e-commerce models It is actually Stitch Fix acting as the personal stylist, choosing the items to send to clients.
So rather than a client needing to shop or browse or filter, We're actually choosing things based on their preferences that we know they're going to love and then making a very literal bet on that recommendation by sending the items to the clients.
And the way it works is you can you sign up online and tell us about your preferences and Once you get a shipment from us, you have a few days to try things on at home, see how it looks in the context of your wardrobe, and then, of course, send back anything that you want to return.
And so, you know, somewhat surprisingly, you know, data and algorithms are actually really at the heart of Stitch Fix and have been for many years now.
And ultimately, that's because we have a really great model of learning about client preferences through what people tell us when they sign up and through feedback.
And so we have a really... a lot of individual data about our clients that we can use to really get to know their preferences in a way that lets us manage inventory and choose choose just the right things to send to clients.
I'm the Chief Algorithms Officer at Citrix and the team that I lead works on applications of data and algorithms. throughout virtually every aspect of the business.
So using this data and the feedback loops that come with the business to really drive the experience the clients have and to continue to evolve it as we get better over time.
I always think about things from a consumer perspective just because that's kind of my background and my mindset and I
I'm currently deep down a sneaker rabbit hole, so I have my own shopping vices.
Let's say so from a customer perspective, I sign up, I answer some questions about my style preferences.
I would assume, you know, my my size and body type. and that kind of thing.
And then it goes into your machine, so to speak.
So you've got algorithms. You've also got human stylists.
What kind of items are we talking about?
And then as I kind of go forward, let's say over my first, whatever it is, two, three, six shipments, How does that data, including my feedback on which items I keep, which items I send back, How does that get fed into these algorithms?
And what happens on your end with the combination of human stylists and... all of the algorithms that you're running?
So it's a great question, and there's a lot to unpack there.
I think one of the most interesting themes about Stitch Fix generally is the combination of art and science and figuring out the right balance of algorithmic decision making and human judgment.
And one of the places, but certainly not the only place where this shows up in our business is in the role of the stylist.
So Stitch Fix employs thousands of stylists who make big selections for our clients equipped with tools that bring them data and algorithms and really It's this combined human-machine hybrid system that's making outcomes get better and better for our clients.
As you noted, when people sign up, you can share your preferences for how you like things to fit and styles that you like and your budget. and really help the stylist get to know you.
But you're also helping algorithms get to know you as well.
As you are a client with Stitch Fix, there's a variety of ways that we start to learn about you and your experience. you know so for example uh when when you get a shipment most clients give us feedback on on everything that we send them And this is because, you know, part of the value proposition is Stitch Fix will get to know you over time and really understand how you like things to fit and the styles that you love.
And then, of course, this data is also very useful algorithmically.
And you can imagine if we send you a shirt and you say that it's too large – we've learned something about you and how you like things to fit.
If we send a shirt to a thousand people and 900 of them say that it's too large, We're also learning about the shirt.
And so we can use that to better make better decisions and sending it to people in the future.
On the inventory and merchandising side of the business, we can actually use that data to potentially just change the shirt.
If it's part of our exclusive brands or to make better decisions about buying more shirts, new shirts in the future.
And so it's really this feedback that powers kind of a virtuous feedback cycle where we do a better job understanding client preferences.
And at the same time, that lets us do a better job managing inventory and making sure that we have everything that's going to delight people going forward.
There's a few ways that people can share feedback with Citrix.
There's, of course, the the feedback that accompanies effects that we send people.
So implicitly through what you choose to keep and what you choose to return and, But also clients can share structured feedback like this is too big or too small, or I love the style, but it was a little too expensive.
And so there's a lot of feedback that accompanies the fix.
More recently, we've been expanding into some other experiences for clients.
A good example of this would be the Style Shuffle app that you can find in our mobile application.
And this is basically a rating game where you can provide feedback on images you like or don't like and give a thumbs up or a thumbs down.
And through this, we can really start to understand your style preferences.
And so this is a fun way of learning about clients' outside of the fix, so to speak.
So a way for people to share preferences beyond just what we've sent them in a shipment.
I'm listening to you describe this, and I'm thinking about AI and machine learning and data science in the context of trying to make better predictions, right? your system is getting to know me and starting to understand that I like things that aren't too tight, but I have a tendency to order things that are too large and then they don't look good.
And so I'm, I'm disclosing to oversharing here on the podcast.
That's all right. And so I'm thinking about it in terms of like, you know, honing in and getting it right for each client.
But you mentioned the style shuffle experience, and I'm thinking that, you know, part of the fun of clothes and shopping and whatnot is every once in a while you want to take a left instead of a right and mix it up and have something new in your wardrobe.
And so from a data science and AI perspective, What's the challenge in that and kind of not just getting to know your clients and making good predictions, but then training algorithms that can... introduce new things that will delight clients.
Yeah, absolutely. And I think you're exactly right that there's a real kind of serendipity and exploration discovery aspect to the business that really makes a lot of fun.
I certainly know that I, you know, have been delighted by many things that I never would have picked for myself if I were browsing in a store or shopping online.
And I do think that's a real component of the value proposition for a lot of clients.
I already have six clients Navy blue cotton short sleeve t-shirts in my size.
And if I go to the store on my own, I will buy another one because I have lizard brain.
Yeah, absolutely. And so I think, you know, the type of feedback loop that I think you're imagining is if we were to notice, say, like, hey, you don't know, I really like those Navy t-shirts and we have to send you more and more.
You know, at some point, You know, we've narrowed too much.
And so I think in the context of the recommendation problem, you can think about it in a couple So I think there's the stylist helping to keep introducing some exploration into the things that we're sending and helping to think about What is the client really looking for?
How does this fit into their wardrobe, so to speak?
I also think from a machine learning perspective, you can think about it a little bit as a question of resolution.
And so what I mean by this is if you look at a very, very granular level that, you know, Noah, you know, likes Navy, you know, does that mean that we should always send him Navy?
Probably not. On the other hand, does that mean that we should send you things only that aren't Navy?
Probably not as well. And it's somewhere in the middle where you start to get a sense of your style and aesthetics that you like that aren't too specific so that it's going to duplicate something that you already own, but also getting to understand that you likely do have preferences and will be better able to serve you if we understand those and intend to send you things that we think you're going to like.
So let's talk for a minute and not asking you to divulge any of the secrets that have led to Stitch Fix's success.
You know, we talk about the human stylists and obviously in the realm of AI and particularly in kind of the...
The larger pop culture view of AI, there's this notion of the robots replacing machines and all that kind of stuff.
You have a model where you've got the machines and the people.
It sounds like working in concert. And you mentioned thousands of human stylists who are working with the clients.
And also I would assume doing things to sort of tweak these algorithms.
Can you speak to anything, not even necessarily about the actual way that gets done, but what you've kind of gleamed or the company has gleamed about this human in the loop algorithm process, if you will, and lessons that can be learned, you know, maybe applied to the broader industry from that.
So this is one of the most interesting themes about Stitch Fix as a company and as a service.
I think the role of the stylist is incredibly important.
And I think, you know, inevitably, invariably when I, when I, speaking conferences or go to different gatherings I've always asked you know is the aspiration really just to automate the stylist away and I think we can very emphatically say that it's not.
We're on a journey of understanding what decisions are best made algorithmically and what decisions are best made by humans with just better intuitive, holistic sense of what a client might like.
So there's examples that fall into both of those categories.
And so I think at a very high level, humans and machines are better together.
And I think this is actually kind of a nice... counter story to the more mainstream narrative about, you know, automation and machines, which, you know, feels very dystopian.
You know, the robots are coming and we're going to we're going to lose. you know, valuable human jobs.
And so by thinking carefully about the way that we combine human judgment, in this case, the decisions our stylists make, We can really create an outcome that's better than both could do alone.
And there's a few examples of that. So our clients, for example, when you order a ship and can write in a request note saying like, hey, I'm going on vacation, send me something for the beach.
I'd really like something fun to this wedding that I'm going to.
And while we use natural language processing and algorithms to try to understand that, It's very difficult to match the human capacity to just kind of intuitively understand the intent. of a client's request.
So that's kind of a language example. I think another place where this shows up is in vision.
So it's a very exciting time in computer vision. and things are advancing really rapidly.
And one of the ways that the clients interact with us Many choose to share a Pinterest board where they can pin just items from around the internet that represent something they're looking for or even just a style that they like.
And of course, you can and we certainly do work on trying to understand what is the content of those images that we can use to improve algorithmic recommendations.
And that gets more and more exciting as these algorithms get better and better.
But again, it's a very high bar to clear to even get to parity with Just the human ability to even just glance at a few images and understand the intent of a client and and their aesthetic.
And I think it's largely in these unstructured types of interactions where humans really differentiate themselves.
And that shows up not just in understanding clients, but informing relationships with them as well.
So We really want to be a partner in personal style for people over a long period of time.
And a lot of our clients find it really rewarding to be able to have their stylist, you know, as a human, get to know them and to learn from feedback that the client leaves and, This is all augmented and complemented with what we can learn algorithmically, but I think there's a really rich human component there that is not something easily replaced by an algorithm.
My guest today is Brad Klingenberg. Brad is the Chief Algorithms Officer at Stitch Fix. a San Francisco-based company that curates style for over 3 million clients.
They send out fixes. boxes of curated style clothing accessories for men, women, and kids.
And they do this through a combination of of, as we've been saying, art and science, human and machine, working in concert.
It's beautiful. Fashion will lead the way.
Away from the dystopian thing that some people think AI is coming and actually to the truth, which will be more interesting, creative and fulfilling jobs for everybody.
Brad, let's change gears for a moment here and talk about you and your journey.
How did you get into working with data and machine learning and AI?
Yeah, so I studied applied mathematics as an undergraduate and typically found problems with data to be the most interesting and exciting.
And I had the good fortune and just some good timing.
This has kind of preceded the amazing rise of data science and machine learning and in so many industries that we've seen over the last decade.
And so, you know, being excited by data and just generally real world problems Decided to study statistics, so came out to Stanford and got a PhD.
And, you know, really, you know, imagine myself being your researcher initially and had some industry internships and kind of caught the bug for working on. you know, applied problems in industry.
And, you know, I certainly never had planned a career in fashion or, you know, even e-commerce, but Through working with some folks like Eric Coulson, who founded the data science team here, Ended up getting to meet Citrix and Katrina and saw some of the exciting possibilities for the role that data could play here.
And as of June, had been at Citrix for six years.
And so looking at those six years then, what, if anything, has surprised you about the role of data, the role of machine learning at Stitch Fix in particular?
You mentioned kind of catching the bug for applied solutions in industry.
Are there things, looking back on these past six years, that have really stuck out to you as, wow, I never saw this use of this stuff coming.
Yeah, absolutely. I think the Amazing effectiveness of humans in the loop, I think has been one of the main things that struck me.
There's so many advantages to working in systems that have humans in the loop.
And I think one example would be if you were to try to provide a statistics-like service without a stylist, you would inevitably be focus an enormous amount of your time worrying about edge cases where, you know, your algorithm, you know, for some reason just thinks that, no, it just has to have that polka dot polo shirt.
And worrying about kind of worst case scenarios.
And I think one of the wonderful things about working with humans in the loop is that you have another signal, so to speak, on.
What's going to be good for the client? And this actually provides something of a safety net for, you know, for algorithm developers that, you know, they're still a stylist in the process who can... augment and even deviate from algorithmic recommendations was the right thing to do.
And so this really allows data scientists and folks on my team to really focus on Things that really dramatically improve the client experience and worry less about rare edge cases that stylists will be able to help us make the right decision in.
There's a lot of virtues to the systems that combine human and machine decision-making.
And then I think, you know, I've also been surprised, I think, when I joined, I started working on the core recommendation problem we have.
How do we help stylists pick things for clients?
In the intervening for years, really the data science team has expanded to work on virtually every aspect of the business.
And I think to a degree that I didn't even initially fully appreciate the nature of our retail model, which is something of a push model where we're actually making bets on what people are going to like and choosing for clients.
It creates a feedback loop that really is unavailable to most retail models.
If you go into a store and you try something on in a dressing room and It doesn't fit quite right or you don't like the style and you leave it behind. almost nobody learns anything.
They don't learn anything about you and they don't learn about whatever you're trying on.
And those feedback loops are weak or broken in a way that makes it really hard to make the experience better for customers or even for retailers who just want to carry better stuff.
And Stitch Fix has all of this available to us just through the nature of the model.
It surprised me how that's supported a very data science oriented approach to virtually every aspect of the business.
The work that you do at Stitch Fix goes beyond just the styling recommendations and what actually shows up in the... or what the consumer sees, I should say, and touches on merchandising, inventory, marketing, forecasting, operations, all aspects of the operation.
Kind of thinking about what you were just talking about in the retail industry and just all this data sort of being left on the dressing room floor, I guess.
How do you see things changing? Do you think that this is the way that the whole industry is going to go?
And I know there are lots of things in here about the... demise of brick and mortar retail, where that's headed and, you know, all sorts of things.
But do you have any sort of insights about, you know, how you think retail as a whole is being shaped going forward by data or even sort of this realizing like, man, we've lost all this data.
Like we got to change going forward. One of the most interesting themes in retail, I think, is the just the ever-raising bar for personalization.
I think consumers in retail, but in many other industries, are starting to expect a useful degree of personalization and recommendation in many services.
Having become used to movie recommendations on Netflix or music recommendations from Pandora or Spotify, And it really becomes important to really differentiate yourself, especially, you know, you have the Walmart and Amazons of the world that are competing on cheapest and fastest.
If the value proposition is something else, personalization becomes potentially a really important part of that.
So I do think we'll see personalization as a compelling differentiating factor for retailers increase, but it's hard to do.
And for us, it's taken a really significant investment in data science and technology to power things.
And I think this is interesting set against like a broader trends of the kind of overwhelming availability virtually anything that you could want through traditional e-commerce platforms.
Thinking about if you were trying to find the perfect pair of jeans, you know, having a thousand that you could choose from at some point is actually harmful, is actually makes it more difficult. than just having someone pick out something that they know you're going to love.
So this idea of curation, I think, is an increasingly important theme and is certainly a central aspect to the Stitch Fix value proposition.
The goal isn't to present clients with an unlimited selection of everything they could ever want and then have them browse and filter and refine and compare and read reviews.
But to actually just share what they want and for us just to give it to them.
In some cases, to give people things that they love that they didn't even know that wanted or they wouldn't have been able to ask for themselves.
And so I think this counter trend to just Limitless availability will show up in a few places, and it's certainly a big part of the Stitch Fix story.
Well, I've got, I mentioned before my wife is a client.
I've got two young sons, one of whom basically wears the same black hoodie every day. which kind of makes me happy because when I was around his age, I wore the same black windbreaker every day.
So, But the other one's a little bit more fashion forward currently.
So I don't know. I may have to check out Stitch Fix Kids on his behalf.
So stitchfix.com is the company, but if folks out there want to know more specifically about The work that you're doing, your teams are doing with algorithms and machine learning and everything.
Are there places online that they can go?
Absolutely. So we have a technical blog at Citrix, so meant for technical readers, called Multithreaded.
But I do encourage people to check it out.
And in particular, there's a post there called the Stitch Fix Algorithms Tour, which is A really fun kind of look at the way that data is used in a variety of different parts of the business.
It gives you a sense of the types of things we're doing with data in many different parts of the company.
Brad, again, thank you so much for making the time to join the podcast, uh, and best of luck with everything you're doing at Stitch Fix and, uh, you know, changing the way that people express themselves through clothing.
Thanks, Don. It's been fun. It's a pleasure.
Thank you. Thank you.
