Welcome to NVIDIA's AI Podcast, recording from the floor of the 2017 GPU Technology Conference. a gathering of the AI faithful here in San Jose, California.
We all know how far AI, and in particular deep learning, have pushed speech recognition. whether that is an Apple Siri, Amazon's Alexa, or Google Assistant.
Our guest on this segment, Syed Ahmed, is directing the power of AI toward another form of communication, American Sign Language.
And what Syed has done is set up a deep learning model that translates American Sign Language into the English language.
Syed, welcome. Thank you. It's a pleasure to be here.
Now, tell us, you're an undergrad. Clearly, you're in the computer science department, I imagine. how did this come about?
Like, how did you think about applying AI to really the issues of sign language and the community of either deaf people or people who need to communicate via sign language.
So it really came by when I was working, when I'm working with the National Technical Institute for the Deaf.
I'm working there as a research assistant under Mike Stinson.
And we are studying how to facilitate communication between the deaf or the hard of hearing and the hearing population.
So in current technologies, how the deaf or hard of hearing is communicating is by typing or by writing their writing their voice in a piece of paper.
So clearly there are technologies available for the hearing such as automated speech recognition. which is facilitating your communication.
You don't have to type so much. So similarly, you see that there is no translation system for the deaf or hard of hearing.
So imagine this situation where you want to talk to your deaf or hard of hearing friend and you don't know sign language so what would you do in that case so so you could revert to texting like you say um but if you if there's no facility for that at the moment yeah Maybe it's something else.
So that something else is that you could point your phone at your friend and while they sign, automatic captions which appear on your phone.
It could also be that you could deploy the system on a wearable head-mounted gear such as the HoloLens. you just see these captions appearing as they sign. just a new form of communication.
So it's like Google Translate but for American Sign Language in some sense.
In the case of Google translate I understand it goes from one spoken or written more likely written language to another written language you're bridging a visual language with a written language.
So what were the problems that you faced and how did you start to go about tackling them?
So in our deep learning curriculum, how we are learning this, how do you map English language to, let's say, French? we are just seeing these as sequences.
So that's the ground behind this work as well, that we are seeing videos as sequences. and English can be represented as a sequence of vectors.
Or let's say, just if you think of it like I am displaying this english language in a three-dimensional space so similarly you can do that with video as well different videos, different videos have frames and each of these frames have different features.
Like when someone signs, They do a certain hand movement.
They do a certain facial movement. They do a certain body language movement.
So all these features, you can say that they are the ABCDs of videos.
Right, but you're taking it from, like you said, three dimensions into two dimensions, like text on a screen.
Yes, you can say that. you are reducing the dimension.
Video is a bigger dimension, whereas you are reducing the video to a dimension which is easier to handle, which reduces the complexity of this problem.
And clearly, I don't want to hold my phone up and if I'm having a conversation, I want it to happen, you know, in real time. in a timely fashion.
You don't want to hold up your phone and then five minutes later understand, okay, that's what they said.
So what did you have to teach the system to do and how did you train it?
So what went there is this data set which was released by CCNY, these professors from CCNY, Zhang et al.
And CCNY is? CCNY is SUNY Brooke College in New York.
In New York, okay. They gathered about 17,000 videos from YouTube and those videos, they gathered the captions from those videos.
And since this approach is a data driven approach, you just feed in this video and caption pairs into this algorithm.
And it learns iteratively what it means to sign let's say hello, what it means to sign let's say goodbye.
And how does the processing happen so it's fast enough to, I mean, it's just fast enough to... to actually have a conversation without too much lag?
So that's still an ongoing problem that I'm trying to tackle.
There are since this video processing has a lot of images going into it, there would definitely be a lag.
And those, Those things we are tackling using, let's say, in TensorFlow, the framework that I used, you can optimize these things with, XLA, this new optimization, new compiler for reducing your graph size.
You can also deploy these things in better machines like the NVIDIA Jetson.
So let's say if we could make our phones faster, which we'll see in the future that our phones will be more faster, it will have more processing power, In that time, we'll see that this won't be a problem.
This lag of processing this video won't be a problem.
I can see that. So when you deployed it out in the wild or even in your lab, what's the reaction been and how has it kind of changed the level of communication between the hearing and those who are either deaf or hard of hearing.
So it's very new right now. We haven't tested this with any of the deaf or hard of hearing people, but in the future we are definitely making some experiments where we're starting with hearing or the experimental setup is like you have a hearing and a deaf or hard of hearing solving a decision-making task.
And the decision-making task is something like you're lost at the sea, and you have to communicate which of the essential items that you're going to pick.
So during that conversation, we record a lot of data which measures that, okay, how much time did it take for each of them to make this decision?
How many words did they share throughout this conversation?
So those metrics give us the ability to characterize how good of a conversation that was.
And that's how we compare these studies that, okay, did it really benefit the hard of hearing and the hearing communication or are we just better off with typing and doing automatic speech recognition.
But your vision is that we have, I mean it's sort of the universal communicator, you know in our pockets and we can pull it out and whether that's somebody who speaks a different language and and by different it could be sign language it could be a language that i don't speak and the other person does we'll be able to somehow find common ground.
Yes, my vision is that you would not be required to even reach out to your phone.
So let's say if in the future our world becomes more augmented and Let's say if we don't even have to wear a device like HoloLens and you just speak and you see words appearing beside your friend's head.
Wouldn't that be great? That would be seamless and that would that would encourage more face-to-face communication rather than going through your device, pointing at your friend.
But right now, since the technology is a little limited, it's just a small step forward that's great and i so i get to sort of pick what language i want to show up um you know and and and go from there like this is the one i'm comfortable with this is what i know best yes that's the that's actually the basis of all these neural machine translation system that whenever you are translating a language to another language, it finds a common language between them.
So I forgot the name of the researchers, but A study came out from Google where they found this, they call this universal lingua, where you find a common language between all these languages.
So I can see that happening. You make these languages ubiquitous.
Pardon the interruption, but we have a favor to ask.
Leave us a review on iTunes, Google Play Music or whatever your favorite podcast platform of choice is.
It helps more people find us. As always, thanks for listening.
Now back to the good stuff. How is sign language as a problem? either similar or different to translating written or spoken language.
Because I can imagine that People don't sign everything perfectly.
Clearly, there's slang that crops up into sign language.
So how do you tackle that as a problem? So right now, I can see that this is one of the bigger problems of deep learning right now is that your model is as good as your data set.
So let's say if I had sampled all these sign languages very strictly, like the best sign language videos that I can get, then I can say that, okay, it will work for if you formally sign.
But to make my model more robust, I actually have to gather all, as you mentioned, all these colloquial sign languages that people speak.
If you want to make it even better, go to Japan, go to India and get what they are signing.
So it really becomes then a data collection problem.
While big companies can do that, us researchers have to spend more time on better algorithms that can actually generate new form of videos, it's still a challenge.
I can't I don't have much knowledge there.
I have to research more. Say you're focused on sign language and it's this translation of the visual to, in this case, text.
But is there is there something else where that that translation from visual to something else could be applied, whether it's gesture control or something.
Can AI, as you see it and what you're working on, go into different areas that way yes definitely so the uh The high-level problem that this video captioning system was derived from is called video summarization or video captioning.
So in that case, so for instance, let's say you want to say, okay, you're watching a movie and you want to find out, okay, where is that scene where I saw this red car? and you don't have to scroll through the whole movie, do a rewind or forward. you can just ask the system, where was that red car?
And it would give you that specific section.
So that application is called video summarization, where you are you're relating the video frames to the individual dialogues that the characters say.
Another application could be detecting pose so for instance a let's say okay there is a very good application there you can you can detect the pose of a person how they're walking and based on that you can say that okay is this person prone to some sort of neural damage in the future.
Because apparently there has been a study which showed that the way you walk can signal a lot of things that are gonna come towards you away in terms of your health.
So in the medical business, if you can create such a system where you're mapping these videos to several conditions that, okay, hey, this is this video where this person is walking with a gate.
He or she might be coming up with some sort of bone osteoporosis or something.
In those cases, in those medical applications, you would really need this kind of video to text systems where you can quickly summarize something from a video. because a lot of information is encoded in these videos.
Interesting. So it's still kind of at a very high level you're translating or interpreting this visual information into something else, whether that's medical information, whether that's a scene in a movie, or in your case, what you're working on is American Sign Language to English.
Well, Syed Ahmed, thanks so much. And we will be keeping an eye on things.
You've got one more year of undergrad and then you're off to what?
And then I'm off to, I don't know yet. Good answer.
Well, thanks for joining the podcast. All right.
Thank you so much.
