Thank you. Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Netflix, TV on demand, no commercials, binge watching, Netflix and chill.
Perhaps no company has changed the way we watch television more than Netflix.
That's due in large part to the ever-changing library of TV shows, movies, and incredible original content the company serves up to its many, many subscribers, me included.
But Netflix's success is also born from the pioneering ways the company uses machine learning to keep customers engaged.
Joining us is Justin Basilico, a research and engineering director with Netflix.
And while Justin can't share any spoilers today, he knows better than anyone how entertainment and machine learning intersect.
Justin, thanks for coming on the AI podcast.
Thanks for having me. I'm excited to be here.
So let's talk about deep learning and recommendation systems. if you've used Netflix, or really if you've gone shopping online, sort of almost anywhere, you've encountered a recommendation system.
It looks at something, your recent browsing history, deeper browsing history, whatever it is, and comes up with recommendations for what you should buy, or in the case of Netflix, what you should watch next.
But deep learning and recommendation systems kind of haven't always been...
But the use of deep learning for recommendation systems is actually a new phenomenon, if I've got it correct.
Tell me a little bit about what's going on with deep learning and recommendation systems.
So obviously when deep learning started becoming popular and having a lot of traction in computer vision and NLP, There was a lot of interest in the recommendation systems community to try to see if some of those techniques would apply to the recommendation problems because there's typically a lot of commonalities between especially NLP problems and recommendation problems.
And what we saw in the literature plus research some of our own experiences internally at Netflix was that the typical deep learning approaches applied to the traditional recommendation system problem which is one of thinking about having a big matrix where on one dimension you have all of your users represented by index and another dimension you have all the various items, say Netflix cases, the videos on another dimension.
And you have some entry in that matrix that represents someone having rated something or watched something or purchased. that applying those techniques just directly to that matrix didn't create that huge gains that you had seen in other domains and kind of performed similarly to the some of the traditional matrix factorization type methods that were used there.
Since then there's, there's definitely been a lot of interesting work and using our recommendations for deep learning, but it wasn't until. recently in the past few years that really there's kind of a paradigm shift where people are really starting to see traction and using those.
If you know, what kind of nudged that paradigm shift along?
I think it was really about just not thinking about the recommendation problem in this very simplistic way of having this one matrix where you're just trying to fill in and predict these missing entries and instead thinking about all of the other ways you could formulate the recommendation problem that might provide additional information and additional signal that it made sense to use deep learning models on.
One of the examples of this is, of course, looking at things as a sequence prediction problem. but not just a vanilla sequence prediction problem where you just have, you take the things, say someone has watched and you make it into a sequence,
But you look at all the other information you have.
So you know about things like what device they're on, what country, what time of day in the Netflix domain. that information lets you kind of unlock being able to make really good contextual recommendations, which is something that you could do previously with other types of methods, but because with deep learning, you can train really big networks and really handle the sequence and the time information together, you're able to have a lot more accuracy.
And so you guys are working with GPUs to train your recommendation systems.
So we have a lot of different recommendation systems.
And for some of them, we use GPU training for the models you know, where we do deep learning.
And those are mostly done with, because Netflix runs 100% in Amazon's cloud.
We use cloud machines that have GPUs in them.
What have you seen, you know, maybe that surprised you or impressed you or opened up, you know, possibilities for you, specifically with working with the GPUs?
So the thing that the GPUs have enabled is being able to try to take these... larger, more complex models and applying them to the types of problems that we have.
Because these models are so compute bound, having Um, something that can really efficiently process them for training time enables us to. do experiments and try out and rapidly innovate on these types of models that if you're waiting you know, an extra order of magnitude of time is just going to make it much slower for us to learn and improve them and also improve delay the time it takes from actually getting some data to training a model and putting it in production to be fresh and have that data.
When you're building a recommender system for a subscription video product, basically, I would imagine there's one way to sort of get a gauge on, you know, are these experiments working?
Are they just sort of in the technical realm?
Are they working well? But how do you sort of measure the success of what you're doing relative to the business?
I would assume you're not trying to drive people to watch a specific show or a specific kind of show.
And, you know, it's a little bit fuzzy to sort of gauge like, oh, subscriptions are up 10% this quarter.
It's because of my recommender system. Yeah, so the way we approach it is through doing A-B testing.
And so when we have any type of algorithm improvement that we want to try to test, we You have some group of people that we have, you experience the current production version, some group of people then experience a new recommendation algorithm, and then we track those two groups.
And we look at long-term metrics. So because we're a subscription company, our North Star is really you know, whether or not people stay a Netflix subscriber over time and we have a monthly subscription.
So that means, you know, typically it takes a month or two to kind of be able to measure that.
But we see that as the ultimate signal that people see enough value to stay with Netflix.
And if we can move that with a recommendation algorithm, we know that we are making people happier and making a better user experience and improving their satisfaction.
Now, you can imagine... We've worked on improving that for years, and it can be a tough metric to move because there's a lot of things involved, but we have...
Um, other things that are, we know are correlated with that.
So things like, you know, how many hours people watch, does that go up or how many days do people come? to visit the site.
Right. Or, you know, more fine-grained metrics.
So my team, works on the algorithms especially for generating the home page so seeing you know where are people finding things on the home page and if we're improving the home page algorithm we should see the amount of engagement on the homepage go up relative to other places on the site.
Well, one way or another, I landed on the Great British Baking Show.
I'm happy I'm deep into season three. No spoilers.
We're talking today with. Justin Basilico, he's a research and engineering director with Netflix.
Everyone knows Netflix. Netflix, the on-demand video subscription service that...
I mean, really has changed the way people watch TV.
And Justin and I were talking offline before we started.
It changed the way that kids experience commercials, too.
My kids watch Netflix. They don't watch a lot of live TV.
And so the first time they saw real commercials, Like, what is this?
And it just kind of speaks to Netflix's pace in the cultural pantheon.
So let's shift gears a little bit and talk about your background and how you wound up leading a research and engineering team at Netflix.
The obvious questions about, did you watch TV as a kid?
We'll leave those aside for a minute. How'd you get into, you know, deep learning and computer science?
Yeah, so I think part of it does go back to like when I was a kid, I was very, you know, I love building things and Legos and robots and things like that and that.
Eventually led to me when I was in college being interested in computer science.
And unlike some of my friends who started programming really earlier, they were always like, oh, you're not going to be interested in. in programming computer science.
But when I started taking my first classes there, I really loved it and kind of was interested in pretty much anything computer science and being able to create things with software.
And this was undergrad? This was undergrad.
And then I happened to take a class on AI. mostly a classical AI class, but there was a part class where we looked at doing a simple perceptron algorithm, which is, you know, the very... you know, basic machine learning building block.
And, you know, I, I kind of, I love doing that.
And then there was a second class in my undergrad on like neural nets and And this was way before deep learning was popular and kind of a time where no one really seemed to actually use neural nets, but I was able to take a class in that.
And then And then I realized that machine learning is really just another way of being able to solve problems computational problems, particularly hard ones that you can't solve by just thinking really hard and coding it out for yourself.
Right. And I saw how much possibility there was of doing interesting things.
So in grad school, I... Kind of focused on machine learning.
And then in grad school, I actually got into doing recommender systems.
Okay. I think part of that was, you know, my wife now, you know, at the time, fiance, we'd spend a lot of time going to, you know, video stores and...
Wandering around the aisles for, like, 45 minutes to try to figure out, like, what to watch.
I remember that. Yeah. you know, there has to be a better way to help people find, you know, something that they're going to enjoy so that they can make the most use out of their time.
And so I actually started doing work in recommenders then.
After I graduated, I worked in the national labs doing kind of just general machine learning in a cognitive systems group and worked in a whole bunch of different domains from you know, doing things with cars and detecting people who are either overloaded in terms of having difficult diving situations or falling asleep, worked on training systems to help people learn better. worked on text analytics and personalized search for conference papers and research.
And then Netflix reached out to me And it took a bit of convincing, but, you know, and it was kind of returning to something that I love doing.
Sure, yeah. I've been at Netflix for about six and a half years.
I started as an individual contributor working in the ranking kind of on the early days of that kind of deciding and trying to predict out of our whole catalog, what are people going to want to watch?
Um, and then, um, kind of moved into a more management role looking at this whole page construction problem that's kind of what brought me to here.
So on one hand, like I started my exposure to neural networks pretty early on.
And then in terms of deep learning internally at Netflix, Of course, when people start talking deep learning, we got really interested in it.
And there were a few attempts to try to where people kind of heard what was out there and said, oh, maybe we can replace large chunks of what we have with deep learning.
And a lot of those projects didn't really pan out.
And so, you know, there's a lot of hype around deep learning.
And so there's this question of like, is it all hype?
Is it real? And so, you know, being involved in machine learning community, like learning more about deep learning, kind of had these insights around, OK, well, maybe it's not about Just to be in like the problem where you're typically doing, but maybe this is a tool that can help us solve problems that we haven't been able to really. in other ways and kind of open up the space of possibilities of what we can do.
So you kind of touched on this a little bit, but what, if anything, along the way... anywhere along the way, or specifically in your six years with Netflix, has maybe surprised you about the direction the industry's going, maybe an unexpected twist your own work took.
I think one of the more interesting things that I learned kind of working at Netflix and on these problems is really about something that people often gloss over when they're doing recommendations and sometimes machine learning in general, which is the importance of time in recommendation and machine learning. systems because a lot of times, especially when you're doing machine learning for real applications in production,
What you typically have is you have some historical data and you're going to build a model and you need to be able to predict future data to do that. and it sounds really simple and straightforward but if you look at you know classically the way people set up data sets like they don't do temporal splits they don't under look at the evolution of these predictions and models over time.
And so what What we've kind of learned is that beyond users and items, time is probably the third most important signal to a recommendation because you're ultimately always going to be using your model at a time that's beyond what it ever saw in training.
You can try to close that loop if you're doing a lot of online learning and things like that, but you're always having to extrapolate.
And that's one of the areas that we saw the potential for deep learning to help with, which is you have... if you have a very discrete model that might understand days in the past or time of day or day of week,
You can do some kind of extrapolation on a cycle, but, but, you know, if you're doing recommendation, there's a lot of popularity trends, you know, globally in specific countries or even within specific interest groups. that you would love to be able to project those out.
And by incorporating time into our recommendation systems in many different ways, We're able to see it's just a super important dimension that you can very easily ignore, but if you can take advantage of it and respect it throughout your whole The way you collect data, the way you train your systems, it makes for much better generalization.
And so when you're saying time, I mean, that can mean anything from I last watched a certain program and how that factors into the recommendation algorithm to kind of knowing, oh, I prefer to watch, you know,
This program or these sorts of programs in the evening versus these other ones in the morning and time of day.
Yeah. So those are different ways that time can impact recommendations.
And like, just part of our lessons like that.
Those are very impactful and it touches these systems in so many different ways that it's very important to just keep in mind no matter what you're doing.
All right. So taking time as a concept and leaping off of it, where do you think your work's headed over the next few years?
And either, you know, again, I'm not asking for spoilers from Netflix.
Sure. What's going to happen in Orange is the New Black season?
What are we up to? Six, seven? But whether specific to recommendations or just kind of more broadly to the areas of deep learning that you're working with, You know, any thoughts on where we're headed over the next five, three years, five years?
Yeah, so I see a lot of different opportunities for how both recommendation and also where that intersects with deep learning going in the future.
One of those ways is we want to go beyond just deciding what content and personalizing those in terms of recommendations, but also personalizing how we recommend recommendations And we've made a few starts at doing that.
And in terms of personalizing what images we show on the homepage, And that's an area where again, when you think about images and recommendations, That's a natural place where we could see some good connections between recommendation and deep learning.
We're also interested in thinking about problems that we have in terms of when we show recommendations to people, and people click on it, are they clicking on them because that's what they really wanted to watch and would be the best thing for them?
Or are they just like, oh yeah, okay, that's at the top, I'll watch it.
You know, our goal really is trying to find the best content for each of our members that they're really gonna enjoy.
And so to do that, it gets into this kind of causal problem of we don't want to just kind of make our recommendations correlational.
We want to like have a good causal relationship set of recommendations we put for our users so that they do find that thing they really love and that our system isn't just kind of trained off of its own feedback loop.
And so that's an area that there's a lot of potential to get better.
And another area is reinforcement learning.
And in particular, you know, where that intersects with deep reinforcement learning to try to again, optimize these longer-term objectives and metrics to understand You know, maybe you got someone to watch something today, but again, maybe that wasn't the best thing for them.
And tomorrow. they don't come back and maybe eventually they cancel or, you know, maybe it was just okay.
So how do we make sure that our recommendations isn't just making that kind of good action for right now, but it's looking at the long-term because that's what can help us deal with challenges of, you know, not just kind of latching onto one interest that someone shares with us in the service, but actually learning about the diverse set of interests that people have and helping them understand all of the great content and all the different dimensions and exploring someone's interests.
We do some of that, but we think by looking at these more longer-term optimization objectives that we can use reinforcement learning potentially to get better at that.
Awesome. Well, again, full disclosure, I'll be watching your work carefully several times a day these days on Netflix.
Obviously, folks know about Netflix. If they want to find out more about the research and the work that you, your team is doing, anywhere else online, they can go.
Yeah, sure. So we have the Netflix tech blog and we're additionally soon going to be launching a research webpage that will have all of the different research areas that Netflix has running and people can find out a lot more about all those different areas and how we're doing. and research across Netflix to make our customers happier, make better content and have a better business.
Excellent. Justin, thank you so much for joining.
I'm biting my tongue. Spoiler jokes. Throw enough of those at you.
Thanks for joining the podcast and look forward to following your work.
Thank you. Thank you. Thank you.
