Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. There's never been a better time to be a sports fan especially if you're into watching highlights and video clips online.
Between broadcast TV, streaming platforms, and endless video content available across myriad platforms, a sports fan with an internet connection is never wanting for content.
But how are all of these video packages made?
Editing together so many social media clips, long-form YouTube highlights, and other videos from global sporting events is no easy feat.
That's where auto-magical video solutions help.
And by auto-magical, I of course mean AI-powered.
Our guest today is Amos Berkovich, Algorithm Group Leader at WSC Sports. makers of an AI cloud platform that enables over 200 sports organizations worldwide to generate personalized and customized sports videos automatically and in real time.
Amos is here to talk about how WSC Sports Platform uses deep learning to turn in-game action into social media-ready highlights Faster than a VAR review in Premier League.
Well, a lot faster than that, I hope. Amos, welcome and thanks so much for taking the time to join the NVIDIA AI podcast.
Yeah, thanks for having me. So why don't we start?
I teased it a little, used some language.
I'm a big fan of the word automagical. You guys have that on your website, so I grabbed that one.
But why don't we start with you telling the audience a little bit about what WSC Sports does.
Okay, cool. So WC Sports is a sport tech company, and actually we automate the creation and distribution, as you say, the highlights.
We can say that for over more than 250 teams, leagues, broadcast partners across the including the NBA, ESPN, the English Premier League and many more.
Actually, we use machine learning technology to analyze Sports broadcast, we analyze and detect each event in the game, each event that occurs during the game in real time.
And then with this indexing information, customers are able to create highlights according to their own demand.
For example, each customer can decide that Every half-time break, I want the system to create a movie with five minutes with the best highlights of the game.
And maybe I want a shortened video that will be maybe, cropped vertical for mobile applications, and I want it to be two minutes long, and maybe I want a longer video with more events and it would be 10 minutes long and I will put it on my app so Without using WC, so you need a video editor for each movie, you need to create it manually, right?
Right. So now the system enables the customers to monetize their most important value, which is their media rights, and they create any content that they would like.
Content could be personal. I mean, take a specific player and just every time he scored more than...
20 points, take all the plays that, for example, Steph Curry was involved with, combine them together to one clip and distribute it to certain players.
For example, we have An Israeli, I don't know if you know, an Israeli player in the NBA.
He plays in Washington, the Wizards. Oh man, it's the off season and my NBA brand is off.
I've been following Premier League. Who's the player?
Benny Avdia. Yeah, yeah, yeah. Apologies to Danny.
Actually, what the NBA can do is like they can make a rule specific for Danny after automatic rule.
And they will say, okay, every time they will make more than four points or I don't know.
Yep. take all the events that he was involved in or just maybe the basketball shots that he made in the game, create a clip and distribute it to your customer rights holder in Israel, which is a sports channel, and send them Danny's clip, and send players from France to a France channel or whatever.
So that's actually enables the customer to create more content and more specific content to it.
So instead of my job perhaps having been, I'm a video editor, I work for the Golden State Warriors. or ESPN or whoever has the rights.
And I want to create, at halftime, I want to create a package of Clay Thompson's first half highlights.
In the old days, I would have to take the footage, manually scan through it, pick out the events I want, Clay Thompson's highlights, manually cut them from the video footage, splice them together, do the transitions, do the music, whatever it is, export it.
But now, if I've got this right, however it works, I punch into the system, I set the parameters that I want Clay Thompson highlights from the first half of the game. like clay thompson best three points of the game just right and your platform takes care of the rest yeah Exactly.
Auto-magical indeed. You got it absolutely right.
That's the thing, but... like now just multiply for like 20 types of sports you can do the same thing that you mentioned just but for basketball and obviously football and soccer hockey and also surfing and golf and so we support 20 sports, more or less, with 200, 250 even customers around the globe, that be leagues, teams, etc.
So I know the answer to how does it work is you're using deep learning, you're using machine learning, you've trained your algorithms to recognize... to detect the certain features we're looking for in the video streams.
You've got, I would imagine... Object detection, so you can tell Clay Thompson from Killian Mbappe.
You've got some sort of recognition for the different types of events going on.
Maybe talk a little bit about. how the system actually works and what you've trained your algorithms to pick up on.
Yeah, sure. So mainly we have three main inputs.
The first one will be the video and the second one will be the audio of the game and The third input would be the data feed, the play-by-play data feed which is available online. uh for almost all sports uh in the world right okay yeah so actually we can say that we have In order to create the context I was talking about, we have to solve three main problems.
The first one would be event detection. We need to detect those events that we talked about, every shot, every drive in football.
So in general, it may differ from sport to sport, but yeah, we developed an action detection or recognition models for detecting specific events, for example, basketball shots or classifying between different types of golf shots.
But this task is even more complicated than just detection.
Action detection may involve object detection and then some classification model or in some places that the action itself is not defined just by one player.
So maybe we'll analyze a sequence of all images.
It could be different from sport to sport, but it's not only that that just detect events, which sometimes could be easy, sometimes could be a bit harder.
But you need a smarter system because you need also a system that knows to ignore irrelevant events.
Right. For example, There's many shots in a basketball game that maybe we're taking after the referee whistle. or a nice example I like to talk about is replays.
For example, If you see a game for the first time with your kid, actually, it happened to me a few days ago.
I watched a soccer match with my kid and... somebody scored a goal and then there was a replay and my kid was like going very, oh, they scored another one.
Right, right. And for us, like, Maybe we can call it like we are pre-trained viewers.
We watch sports for years and our mind is like well-trained. to recognize we know that after a goal, a replay will come.
They showed 10 replays, yeah. So, but if you train a model that just detect goals or detect basketball shots, it will recognize the shot and then it will recognize another shot.
Right. So how should the system will ignore those replays or shots that are taken after the whistle? after referee whistle.
So you need to integrate not only just the recognition model, but you also need a system that understands the context of the game.
And this is something we focus a lot on learning patterns of broadcasting, of professional broadcasting.
All our customers are professional broadcasting.
There is stuff like analyzing the box score of the game. if it was a true shot and if the basket was made.
So the score will obviously change. And analyzes graphics.
There's tons of graphics during the game that the broadcaster use, and that helps you as a trained viewer to understand that the replay just happened.
Usually it will be like a graphic transition and then replay them.
Right, right. And go back to, but as a viewer, You don't even notice it, but your brain is like well trained.
But you've got to train the platform's brain, so to speak, to understand, yeah.
Yeah. And amazingly, it works really good.
And we have a system that understands replays, understands watches. understand how to ignore irrelevant actions and how to detect the real ones.
How long has the company been doing this?
How long have you been in operation? from around 2012.
Okay, so a decade, give or take. Yeah. They changed the goal of the company.
We can say that working on highlights from 2014 or something.
Okay. So we'll talk a little later about the differences between sports, as you mentioned before, the context of, you know... golf and individual sport versus something where there are lots of players all in the frame at once.
It's got to be challenges. What are some of the other main challenges that you face in doing the work?
And then maybe what are some of the tools, some of the solutions that... that really help out?
So one of the challenges is first recognize and understand which events are relevant and not.
And here, yes, we will have like classification, maybe object detection alongside with that, classical classification models or sequence and classification models.
Another main challenges would be after you find the event and recognize there was a shot, You need to trim the clip.
You need to cut it as good as you expect a video editor would have done.
So, You want the clip to be as tight as possible.
You don't want to make long clips. You want them to be tight, but also you need them to be informative so you will understand what just happened.
Right. The professional term that in academia, you can find a lot of works about it called Temporal Action Localization.
Like those are models that are trained on somebody's doing action and the model aims to predict the starting point of maybe somebody started dancing and ending point where he stopped dancing.
For us at WSC, we take the temporal action localization like architectures, but for us, the problem is clip trimming.
Yeah. Maybe the action just ended. There was a nice dunk, but after the dunk will come maybe a nice close up. maybe the broadcaster will describe the event in some... Right, he's got a good tagline you want to get in there so you don't want to cut him in the middle of a sentence while he's going oh my god LeBron is having a great yeah yeah and then you can cut it So you want to take into consideration both the audio as a pattern, it's a very important pattern, and also the if the broadcaster is talking or if he's not talking.
And also the video content, not only the action, action start and action end time but also like what is happening if if it was a nice close-up or the crowd or maybe there was a cut for commercial and you need to break really really fast And there are things that you need to leave out from the clip, no matter what else is happening.
So here we need to perform some fusion between the video signal and the sound signal, obviously.
Also, training the system is not that easy task because this task is subjective.
I mean, there's no one starting point and one ending point.
Sure. You can select multiple starting point and end point and both of them would be fine.
So training those models and evaluating them, it's challenging.
And obviously it works really good maybe in some sports and other sports it could be like a bit harder.
Yeah, so let's get into that. Well, I guess kind of two questions.
Are there sports that are harder? What are the hardest sports to work on?
What are the easier ones? And then do you and your work have a personal preference for working on certain sports over others?
Yes, that's a great question. So actually, I think that you can say in general that sports that are easier to trim, would be the sports that are well-structured. sport like like tennis every event in tennis like you have a obvious start point An obvious endpoint where the phone went out or hit the net,
And even the subjectivity would not be that big.
I mean, if we take three different editors, there's a good chance they would agree where the clip should start and where the clip should end.
Whether if you take basketball or soccer, I mean one can argue where should uh the clip should start maybe three passes before the goal maybe five passes that was so subjectivity is bigger, the variance between three manual editings will be bigger in those ports and And the violence would be probably smaller in those sports.
Having said that, doing a mistake in the structured sport is more like mistakes are more obvious in those kind of sports.
For example, if I take golf, golf is like, again, very simple to define what is the starting point, what is it like before the hit and when the ball stopped moving or something.
But if I will make a mistake and I will insert part of another golfer to another golfer clip, that will be really bad for the viewer.
As for maybe in soccer, it's not that bad if I entered some other passes went in to clear off another content.
I'm speaking with Amos Berkovich. Amos is the algorithm group leader at WSC Sports, who are...
Providers of an AI cloud platform used worldwide with professional sports organizations to create automatic Video clips highlight packages from real-time sporting events.
We're talking about how... AI, deep learning, machine learning, powers the platform and takes what really, if you've ever done any video editing, you know it can be a very long labor-intensive process to trim those clips just right, get the sequencing down, get it all packaged up into a professional watchable package.
And they're doing this automagically with their platform.
Amos, we've got a couple of minutes left here.
Are there... Things about the process, about what WSC does, whether it's specific to one project, one task, or kind of more broadly, that might surprise or amaze the people listening now and thinking about all the technology, all the magic that goes behind the scenes to automate this process.
Well, there are tons of things I think may amaze people, but I think one story is when we started working on it, we didn't realize how a huge impact that model might have. would be we have automatic tool crop product that customers can create, they can choose if they want the clip to be like in regular format or it could be in vertical format.
Right, right. Everybody likes vertical video now.
It used to be nobody made, you know, you wanted... you wanted the wide format to get more action, but with mobile devices.
Yeah, you watch soccer, you want the wide format, you want to see what's happening, you want to see all the players, but apparently, and what amazed me that, I'm not a business person, so I'm not sure about the numbers, but it amazed me how huge was the demand for crop, for vertical cropping.
So when we started, one of the guys in our team, when he started working on this, we didn't thought the impact that would be for such a tool because we train the system not just to track up after the ball because Because you don't see just the ball while you're watching the broadcast.
You have close-ups, you have crowd images, stuff like that.
We train a model on the most interesting point in frame. works really, really good.
And actually, it outperforms the manual labor.
A person cannot crop an entire grain in a vertical aspect ratio and it will not look good in the automatic system. looks amazing.
And also the demand for it from our client side, it's like super amazing.
Skyrocketed. Yeah. Yeah. Another thing that, uh surprised me again during like developing of a system that you think might be complicated and sometimes you have surprises things are not uh as hard as it seems so again some sports And when you come to try to detect how interesting was one event, comparing to other events.
So in some sports, Just by looking at the data, I mean, how much a player how much yards did the player run or how many, how long would the ball went in golf or stuff like that.
So, yeah, Just by looking at the data, you can have a very good ranking model.
And in some sport, the subjectivity, again, It's just huge and a layup in basketball could be lame or could be amazing.
Right. yeah and here you cannot use just data you need some supporting models that may analyze the the crowd cheering the enthusiasm of the broadcaster or some maybe words that were were said, so it's really nice to see how some sports you can solve this problem really easily just based on dry data, on tabular data, you can understand, oh, this event, this is those and these are the best events in the game.
And in other sports, it's like No man's land.
You need more information taken by deep learning tools. to even try to solve this problem.
So let's wrap up on a future looking note.
What's next for WSC? And then if you like, What do you think is next or where do you think the industry is headed kind of more broadly? with the production of real-time video content and the automation factor as pertains to sports or more broadly, if you like?
So I think a very challenging task would be, We give best highlights in a very good way and we know how to spot the most interesting events and bring them to a one. clip that shows the best sides of the game.
But I think telling the story of the game, it's something that we are not there yet.
I think that would be super interesting to see how can we solve this in the next few years.
Maybe a player just made a lame layup, but that was his first layup.
After coming back from injury, right? Yeah.
Yeah. So I remember Clay Thompson. We talked about Clay Thompson.
Yeah, yeah, yeah. I'm a huge fan. Yeah, so when he scored, we made his first basket after two years outside.
So this is like... The story. Super interesting.
And obviously, you want those clips... in the game highlights but like the system say it's just a regular shot and there's no reason that the system will prefer this kind of So I think the storytelling and how to integrate information surrounding the game, like from media, media posts or whatever. that like before the game or after the game to see how you can combine this information to tell a better story.
Super challenging task, but I think it should be interesting to approach.
Yeah, no, for sure. That's awesome. So Amos, for listeners who would like to learn more about what WSC Sports is doing, maybe check out some of your work and then also maybe for some of the technical minded folks in the audience who might be interested in more of the developer-focused, technical-focused stuff that you guys are doing.
Are there any URLs, any places online where you would direct folks to find out more?
Yeah, sure. So we have our website, wcsports.com, and over there you can find links.
We are... active in Twitter, LinkedIn, social media.
So all the links are there in the website and you can follow The Twitter account.
Perfect. Love it. Last question. Favorite sport, favorite team, favorite player, anybody you want to give a shout to?
Favorite sport? That would be basketball, NBA.
Also, I like tennis. Oh, yeah, yeah. You watching the US Open as we record this?
Yeah, yeah. Some great games. Yeah. Favorite play of all time?
Yeah. So in the NBA, my favorite team is the Utah Jazz.
Oh, okay. I saw him last in the finals 30 years ago.
Way back, yeah, yeah. I fell in love with the underdogs, so John Stockton and Karl Malone.
Okay. My favorite, I think, in today's game would be Nikola Jokic.
Sure. Yeah, he's amazing. Well, if you make it to the Bay, if I can find tickets, we'll hit a Warriors game together.
Pick out the highlights in real time. That would be great.
There's a lot more I'm sure we could cover.
So maybe we'll have to get you back on the show down the line.
And, you know, when you guys have that storytelling component perfected and automated and it's making all of us laugh and cry and sing as we watch the highlights.
But this is fascinating stuff. I told you before we started, I'm a sports nut.
I've always been a big sports fan. So very cool to... to talk and hear some about the work you're doing.
And what can I say? It's almost NBA season.
So best of luck. on all your endeavors.
And I look forward to watching your content in the coming months.
Yeah. Thank you very much. Thank you. Thank you.
