Welcome to NVIDIA's AI Podcast, and we're recording from the floor of the 2017 GPU Technology Conference. a gathering of the AI faithful here in Silicon Valley.
We all love a good highlight reel, the spectacular passes, buckets, tackles, goals. whatever is particular to your favorite sport.
And as the old song goes, we love the agony of defeat.
We love replaying all those things that went horribly wrong.
The same thing applies to fans of esports.
Games like Starcraft, League of Legends, Heroes of the Storm.
Fans of those games want to see the kills, the strategy, the moments that are most dramatic.
But the way our viewing habits work these days, especially in esports, There's no time for broadcasters or media companies to painstakingly craft the game's highlights or the day's highlights.
We want it now. Yahoo's esports team is using AI to make that happen, to get the highlight reel from esports out there as fast as they can.
And they use AI to do it. to tell us how we have Yale Song, Senior Research Scientist at Yahoo Research.
Yale, welcome. Hi, thanks for inviting me here.
I'm so excited to be talking this podcast as well as being in GTC, which is very large conference. as you said, the gathering of the AI experts.
Well, you are an expert in image recognition.
I know you did a PhD at MIT. How did you guys at the Yahoo research team think of applying AI to the to the problem of getting highlight reels for esports.
And what are those highlight reels? What are people looking for when they're watching those highlight reels?
So the project all started by having this platform called Yahoo Esports where we have about a dozen of game titles and Whenever there is a professional league matches, then we livestream, we show videos of those.
And as you can imagine, it's not just a single game, it's multiple games.
And it's to the point where we thought, are we going to hire multiple editors for each game title and have them work all simultaneously?
Or are we going to invest in AI and try to see whether computers can identify interesting moments in those video games.
You and I might not agree on a player that we want to watch, a moment in a game that we both like.
It seems like a very subjective thing trying to figure out what's a highlight you know just across the board because what appeals to you and what appeals to me might be different so it's a very human thing How do you get AI systems to start to recognize that?
Yeah, that's a very good question. And I got the question really a lot.
The way we tackle this is actually trying to remove the subjectivity. out of the task?
And that means, is there a certain parts that we can objectively identify as Well, typically these types of events in esports games leads to highlights.
And what's great about esports is it's all computer graphics. whenever there is some interesting dramatic events, then The game makers, they used some visual effects to kind of emphasize what's going on in the scene. the burst of fire, thunderstorm, or even something very explicit like You literally write down some sentence.
Red team kills, triple kills. So those are what we believe to be some objective components that leads to the typical highlights.
So that leads us to thinking that Alright, it's a visual effects and that's great for computer vision.
We analyze pixels from the scene. And whenever there exists those types of a, you know, fireball, you know, I don't know, three swords go up in the air, whatever it is.
So do they always have these dramatic flourishes when there's a scoring moment or a... a moment that changes a game?
Some games do, some games not. Starcraft, League of Legends, Heroes of the Storm, they typically have those visual effects.
Some others are not. Counter Strike. They do have it, but it's a little more subtle.
So what did you teach your system to be good at then?
And how did you train it? I mean, clearly, It's watching games, and did you train it on millions of hours of gameplay, or how did you go about doing it?
Yeah, so that's exactly what we did. We selected a few game titles where we believe there exist very pronounced visual effects.
And for each game title, we collected a little more than 100 hours of videos and not just videos from anyone playing it, but actually the pro league videos.
And we went on to what we call an iterative annotation process, where instead of annotating all those every single seconds of videos, we first annotate perhaps just 10 hours or a little less than 10 hours of videos from scratch. then we go into this iterative step.
And sorry, just to back up, when you say annotated, your people on your team annotated them to Say like, here's what happened and here's what was going on at this moment.
So the annotation connects to the pixels, essentially.
Ah, yeah. So to explain that, I would like to first explain what types of labels that we are talking about here um it's a esports live stream that's our input video and As you can imagine, it's not just all about game interface scene.
There should be some studio commentators, interviews with game players, audience excitements, some ads, etc.
And we first narrowed down the problem into Let's first filter out anything that's not related to the game that we are interested in.
We are talking about Starcraft, then let's Forget about anything that's not related to StarCraft.
So we annotate videos in terms of the scene type labels.
And that's objective thing. Probably you and I or anyone in the street can identify, oh, this is a StarCraft scene.
This is studio scene. So for that task, we used our own editors, Yahoo Esports editors, to go through the videos. and identify this is the game scene, this is not.
Right. Here's what you want to look at and here's what you can ignore.
Exactly. And that's objective task. The tricky part is the subjective part, highlight or non-highlight.
I said at the beginning that we are making a decision based on the visual effects. but it's painstaking to define every single of those visual effects.
And it's, It takes hours and hours to watch the videos label, oh, this scene contains burst of fires, this scene contains thunderstorms.
So instead of doing that, we turn to asking crowds to give us, do you think, this is a highlight or not.
If it is a highlight, what level of highlight is it?
Is it okay highlight? Is it like, oh my god highlight?
And the hope is that, well, People in the crowd, they will make their decision based on whatever they like.
They might make the decision based on the visual effects, exactly what we wanted.
Some might actually... appreciate more on the storyline.
Yeah, or the gameplay. Maybe he or she did something that nobody had ever done before or something like that.
Exactly. But our hope and we have also some empirical evidence that they actually agree.
There is some statistical measure it's called the inter-rater agreement, and there are multiple metrics to measure that, and the one that we used is called Cronbach Alpha, and that's a number between 0 and 1, 1 being perfect agreement, 0 being just random answers.
The level of agreement that we achieved is 0.9, which is really high for this type of subjective task.
Which implies that it's not a subjective task at some level, it's an objective thing.
So once you know that, then do you get to take those moments where there's an agreement and kind of, I don't know, codify them or establish those as, okay, here's what you're looking for.
Yeah. So it comes down to that. All right.
You have hundreds of hours of videos. You have labels, both in terms of the scene type and then the highlight. then how do you go about using AI to solve the problem, right?
So there is this very interesting resource that came out of MIT where It's an image categorization task.
Given an image, give me a single label. And then the data set that they trained upon was It's a natural scene like living room, bedroom, kitchen, church, etc.
It's a very typical task nowadays. It's like ImageNet in different label space.
What was interesting in their research is that they actually tried to dissect the deep neural network that they were training and try to visualize what those inner filters are actually learning, where they are looking at in the picture to make their decision.
And what they have found is really interesting.
For example, for bedroom types of scenes, the network is making its decision based on certain parts in an image that look like this is a matrix, this is a little lamp.
For kitchen, it looks for what looks like a sink. for living room, it looks for sofa, etc.
So even without giving you the region level labels, like here's the bounding box, here's the sofa, Here's the bounding box of lamp.
Even without doing that, the network just naturally learned to make its decision based on those region-level details.
And it's in some sense, it was looking at the highlights of a bedroom or a kitchen, you know, as opposed to like it was staring in the corner of a bedroom and like Isn't that fascinating?
No, it was staring at the mattress or the lamp, like you say, which are, in this situation, the highlights of a bedroom.
Right. So that translates to in our problem, coming back to my explanation at the beginning that we ask the annotators to give us the label, is this highlight or not?
And we have the belief that The visual effects, burst of fires, they will play a very important role in making those decisions.
But we don't want to ask people to give us bounding box of every single of those effects.
Does it sound familiar, Dan? You have very high level tasks. network, you figure out whether it's a highlight or not and I believe you will make its decision based on those visual effects.
Leave us a review on iTunes, Google Play Music, or whatever your favorite podcast platform of choice is.
It helps more people find us as always. Thanks for listening now back to the good stuff.
And so did it work? I mean, as you built out your system, as you built out your deep neural net, did it, I'm sure it got things wrong initially.
And it was, you know, in the analogy, staring off into some, you know, hallway, you in a game where there was nothing going on and like, hey, that's a highlight?
Or did it work pretty quickly? It actually worked pretty well.
So... The way we evaluate it is based on our own data set.
We ran some very extensive experiments because we wanted to roll it out for production and we got satisfying results.
We got the average precision rate close to 90%, recoil rate close to 80%, which is pretty good.
Wait, what's the recall rate? Recall is a measure that gives you, let's say it's a binary classification task, you had out of 180 items that you wanted to detect and let's say you detected 40 as the positive well, highlight in this case, then your recall rate will be 50 because out of not 100, but 80, which is positive, you identified 40.
Precision on the other hand, it measures from different angle.
Let's say, alright, you identified 40 positive items, but On top of that, you also identified 10 negative as positive.
So in total, you identified 50 samples as positive. out of which 40 was actual positive then in precision it's it's 80% because 40 out of 50 was actual positive.
Whereas in recall, it's 50% because 40 out of 80 was identified.
But the bottom line is it works pretty well, right?
And it must be getting better as it's As it sees more examples.
Yeah. So you're saying it's in production now, and if we go to Yahoo Esports, do we see highlight reels that your AI system is... ginning up so there exists a little section in esports called match highlights that's where we show the results So the way it works in real time, these are live streams.
As the game plays, we have a bot that identifies those highlights. and we save little chunks, 10 seconds, 15 seconds of those highlights.
And then right after the game, we only have to look through those small chunks and then add some nice visual effects like what teams were playing, what was the final score, and then create this about three minute video file that we call as a match highlight.
And can you tell me that process versus okay I'm a human watching this I'm gonna edit out you know the highlights and put it together how how quickly does the AI-enabled process happen versus kind of the old way of doing things.
The old way of doing it typically is after the game ends, you have a team of editors watching games again and again and chop it out, some interesting moments, then you pull it together.
So it takes as long as the game happened.
But in this case, in our case, We don't have to do it.
There is a machine behind looking at the scenes in real time and chopping up interesting moments.
Let me ask you this. You've been talking about esports, but clearly it could apply to all kinds of things.
How else do you see this kind of? deep neural net or this kind of training applying to, I mean, certainly to other sports, but are there other scenarios where this kind of deep learning system could do some work.
Yeah, so I did a research project about a year ago where It's a similar task.
Given a video, we want to identify a GIF-able moment, like animated GIF.
Some parts, very short snippet that people usually take as animated gif.
Right. The solution is a completely different system, but what's really different is In esports project, we've been having this assumption that we know exactly what domain we are looking for.
It's to the level of detail of the game title.
We have a model for Dota 2. We have a model for Heroes of the Storm.
On this other project, on animated GIF, we have no assumption about what input video you're talking about.
Our goal is to identify interesting moments so that when it is converted into animated GIF, it makes out a nice animated GIF.
Well, and more than that, is it one of those gifs that goes around the world immediately?
I mean, what you're talking about, too, is the notion of virality.
Yeah, so let me give you a little bit more detail about it.
So there exist many websites nowadays that allows users to create GIFs on their own.
Typical process, you provide URL, YouTube URL or any other video website URL the interface pops up the video and then a little editing tool below giving you the opportunity to create sort of boundary where to begin where to end and then it say well it chops up that part and then it saves it as animated GIF and and there is this whole forum of user created GIFs.
What we did is take all those GIFs and then All those gyps, by the way, have the source for original videos.
That gives us a notion of, well, given this video... Which part did they zero in on and decide to edit it into a GIF?
Right, exactly. Yeah, so we collected hundreds of thousands of those examples from multiple websites and trained a deep neural network where the objective is, alright, you network, you're going to predict a scalar score given a chunk from a video. that whenever there exists a chunk that user selected as animated GIF, that score has to be higher than the other parts of the same video.
This is what we call ranking system. So by training that, we were able to create this system where given any video, you create a GIF.
We actually created a website where you similar to any other gif making websites user provides url and then the machine instead of asking users to define starting an endpoint where it will turn to GIF, we just automatically pull about five to six segments from the video that we believe is good for GIF.
And users can share that on Tumblr, on social networks.
So how do we find this GIF-making machine?
Well this is a research project. This is not something productized.
So I have a link in my website, personal website.
There is a paper called Video to GIF. And there's a link where you can click it and then try it out.
Okay, cool. I mean, and it's not all cat gifs.
I mean, is it something else? Some of them are sports, some of them are cats, yeah.
Well, Yale Song, we really appreciate you joining the podcast.
This is fascinating stuff. And it sounds like there's a much broader application.
What you're describing is this kind of shift from our own notion of what's subjective to what turns out to be objective.
Are machines... getting better and better at that or is there some kind of trickier subjective part that remains and that you guys are you know not stumped on but that That's the human part of things.
I think the machines are getting better at emulating what humans would do when there is very clearly labeled data set.
Some of them might be objective, some of them might be subjective, but at the point when the machine sees it, then there is at least at the current level of technology, there's no notion of subjectivity and objectivity.
There exists this notion of variability.
It's all about statistics and probability.
Having said that, The notion of subjectivity and trying to emulate how humans would think in novel situations, that's something that all AI researchers are trying to solve.
Well, Yale, we really appreciate you joining the podcast.
We'll look for some highlight reels and love the application of this more broadly.
Thanks for joining. Thanks for inviting me.
Thank you.
