Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Today we're looking internally, if you will, speaking with one of NVIDIA's own research scientists about her latest work and her career in computer vision and machine learning in general.
Shalini Demillo is a Principal Research Scientist at NVIDIA with a focus on self-supervised and few-shot learning, 3D reconstruction, viewpoint estimation, and human-computer interaction.
During her time at NVIDIA to date, Shalini has invented technologies for viewpoint estimation learned propagation, gazed estimation, 2D and 3D head pose estimation, hand gesture recognition, face detection, and video stabilization. and I'm sure I've left a few things out.
We're going to speak today about a recent poster Shalini created for last year's NeurIPS conference, that outlines a way to create animated avatars based on video.
And I'm sure we'll get to some other things as well.
So with no further ado, Shalini DeMello, welcome, and thanks for joining the NVIDIA AI podcast.
Thank you, Noah, for having me. It's a pleasure.
So that intro was quite a mouthful, and I know that I only hit on some of the areas.
So I'll start by asking you to tell the audience.
What do you work on? Tell us about your area of research and some of the things that you're focused on right now.
So I'm primarily a computer vision researcher and obviously computer vision and machine learning really go hand in hand at this point in time.
But what I'd really like to do is I like to use my knowledge and skills of AI and computer vision to try to solve real-world problems. that can have an impact on people's lives.
That's really what inspires me at the end of the day.
To put it more concretely, for many years, my research at NVIDIA has been focused on human-centric vision.
What that means is trying to use computer vision technology to better understand human behavior, whether it is trying to understand, as you mentioned, detecting people in images, beginning with that to going on to trying to understand how their heads are oriented. or which direction their head is pointed in.
Then moving on to also trying to understand what sort of gestures they're doing and how their hands are moving.
And then even finer grained, trying to understand how their gaze or the eyeballs are oriented and like which direction specifically their eyes are looking at.
And all of that has led to what I was inspired by was in trying to understand the behavior of human drivers. in cars and I was inspired by trying to create technology which can monitor drivers, especially when they're not paying attention while driving.
Because if you look at statistics, Worldwide and even in the US, 80-90% of accidents have some kind of human involvement. or some kind of distraction.
So that was what sort of motivated me for many years to look at this problem of human-centric vision.
I was out for a run yesterday, actually, as we record this, and Noticed a driver talking on a handheld cell phone while they were driving.
And I know it happens, but I just thought like, wow, people still do that.
With all the technologies hands-free and everything else, that issue of distracted driving is still a thing.
Oh, absolutely. A couple of years back, there was a study at Virginia Tech where they collected actual driving data, crash data, from I believe several hundred or thousand accidents in the United States and looked at them retrospectively.
And 80 to 90% of them had some kind of driver distraction.
Yeah. Yeah. And I, you know, I'll I won't throw stones in a glass house.
I get distracted. I haven't been driving much this year with everything going on, but it happens.
You know, I understand. But yeah, it's a thing.
So you've been working with NVIDIA's, is it called Drive IX?
That's right. Yes, I have been. So you've been working with NVIDIA's AI-based automotive teams over the years.
How long have you been focused on driving-related work?
Right, so since 2013 was when I started working on this area specifically.
When I first started at NVIDIA Research, I wanted to work on automotive vision.
And I noticed that while the maturity level of the technologies for perception of the outside of the car was much higher as compared to algorithms designed for looking inside the cabin.
So I decided to focus my research solely on that.
And so in the initial years in 2013, we were mostly inventing technologies.
And at the time there wasn't a focused product group at Nvidia.
But we kept inventing things and we kept showing the promise.
Then eventually in 2016, you know, a focused product group called the Drive IX came into being.
And part of it was inspired. I like to think that it was inspired by the research that we were showing, which was what was capable.
For sure. So what are some of the hurdles that you face when... you know, working on the technologies and the algorithms for detecting, you know, where a driver's gaze is or how their head is oriented, all of the different things you mentioned.
What are some of the hurdles Speak to them on their own, but I'm also curious, as compares to what engineers who are working on that exterior-facing components of driving technology are facing.
Well, it's hard for me to compare, to be honest, because it's not an apples to apples comparison.
They're two different problems, right? Sure.
The other thing is that I never really focus too much on the exterior side.
I don't quite I mean, I have a high level understanding of the challenges that they faced.
Right. But I can certainly speak more to the challenges of the problem that it's a separate.
Right. It's a different problem. So. but I can speak more to the challenges that we had to solve.
So I think one of the big challenges for human-centric vision in general is that you know, one size doesn't fit all.
So there's often a need to customize solutions to every subject for that matter.
So to give you an example, there's various examples, but the way that you move your hand, the pace, the speed, the shape of your hand, everybody's different.
If you look at more fine-grained problems, like for example, gaze estimation, your interpupillary distance or the distance between your eyes is different for every individual.
If you want to do very accurate gaze estimation, You actually have to deal with the fact that your eyeballs are slightly differently shaped.
Your corneas have slightly different refractive indices.
So you're anatomically different, right?
So the one big challenge in sort of human-centric vision is that one solution does not out of the box fit everybody.
There's often, you'll get to a certain level of accuracy perhaps if you try to devise one solution for everybody.
But if you really want to get very accurate, you have to find a way to sort of customize and personalize your work. deep learning or your AI solutions to each person.
That becomes challenging, especially with deep learning where deep learning typically involves these large convolutional neural network models that have many parameters and you can't simply It's harder to just have, let's say, a few examples of a person and be able to just train them with those few examples you're seeing in large models. it becomes tricky and you have to find ways to do few-shot learning on them to customize them for everybody.
And what is few-shot learning for listeners who might not be familiar?
Right. Few-shot learning in comparison to your classical learning is of deep networks is this idea that you be able to train your neural network with very few examples and by very few it means oftentimes less than 10, for example. labeled examples of a particular concept or person or whatever have you.
The challenge there is that When you have large learning AI models, If you try to train them with very few examples, they will simply overfit to those examples.
And by overfit, what that means is they'll just memorize those specific 10 examples.
Because they have so much capacity to learn that they'll just precisely memorize them.
Then what happens is that if you present that system that's been overfitted to those very few examples, With other data from the same subject, it won't generalize to the other data.
It will not know what to do with that other data.
Because it's just like it's just like fine tune on those 10 things.
And it's like that's those 10 things is all that it's learned.
Yeah. That is why typically for training networks that are larger, you need more and more data.
The challenge in FreeShot Learning is how can you actually train the models with such few examples in such a way that it doesn't memorize, but instead actually sort of learns concepts about that person such that it can solve for new data of that person that it receives.
And so I'm assuming that... If it's an area of work that's been named, there must be some progress being made, but it seems so counterintuitive to everything that We've all come to understand about more data is better.
So how do you get around that? Right. So that's really interesting.
I think for me, there's certainly a progress being made in this area.
There's a specific area of AI called meta-learning, which is learning to learn.
So the idea being that you actually learn the algorithm, which learns to learn from few examples. which is kind of very promising and it's being used quite a bit for few-shot learning.
Metal learning is an approach people use to do few-shot learning, however, The algorithm by itself, when you have such less data, you're working with such low data regimes, you can quickly get into instability with training and things like that.
Oftentimes, what ends up happening is you cannot apply these algorithms out of the box, meta-learning algorithms out of the box, you have to introduce some domain knowledge about your problem.
That could involve, for example, trying to embed your information in some sort of space, which is lower dimensional, so that you have some hopes of being able to have meta learning learned successfully. or work successfully.
But the idea being to basically distill your information down into some lower dimensional space is really the idea.
And what that lower dimensional space needs to be depends on each problem.
And that's where your knowledge of computer vision of the problem that you're trying to solve comes in, of finding that lower dimensional space.
So let's shift gears. No terrible driving pun intended just there.
And talk about NeurIPS. We're recording this in mid-January 2021.
I'm happy to say 2021 for many reasons. But just about a month ago or so, NeurIPS took place in December 2020.
And you had two posters at NeurIPS, I believe.
Is that right? That's right. So can you give us a little bit of a summary for each of those?
Sure. Yes. So I had... One poster that was titled online adaptation for consistent mesh reconstruction in the wild.
And then the other poster that we had was on self-learning transformations for improving gaze and head redirection.
Okay. So both of these posters had one underlying common thread, which was in trying to think about how one can learn either without labels or with let's say less supervision or less labels for two different computer vision tasks.
And I'll sort of elaborate more on that.
So the first one where we were trying to do consistent mesh reconstruction, it sort of uses this idea that we as humans, like when you look at an image, even let's say an animal or object that you have never seen before, you intuitively think about what its shape might look like.
Even though you're just like looking at a two dimensional projection of that object.
Right. And you maybe cannot see the back of it, but you imagine it in your head.
Maybe you look at the front faces and you'll think of what its 3D shape would be.
So this first paper that does the online mesh reconstruction is really interesting in that it uses this, it trains a neural network from just a bunch of images of let's say birds.
So you have like a collection of bird images and all you have is the silhouettes of the birds annotated in those images.
And then what it tries to do is it tries to learn how one could reconstruct the whole bird and the shape of that bird by just looking at 2D images and never having like 3D models of the birds, right?
Which is very intuitive to me. Humans can do this, and so why is it the machine learning algorithms cannot do this?
So in this particular work, what we observed was, so we had a work preceding this one, which we published at the ECCV, the European Conference of Computer where we showed that we could actually do this with, you know, if we had like a video frame, we can do this frame by frame.
But then when we took that algorithm, which operates frame by frame, what we saw was that the mesh that gets reconstructed is sort of, it jitters from frame by frame and it keeps changing in shape and it is in a consistent shape across the whole video.
Temporally, right? But that's what you would want it to be.
So in this particular work at NeurIPS, what we investigated was how can we now try to use this video signal temporal signal and try to have a more coherent reconstruction, which does not change unstably temporally.
And what we found was that There's a lot of evidence that you can essentially get across video.
For example, so as part of our algorithm, we estimate the shape over time.
We also estimate a segmentation of the rough parts of the object over time.
And then we also estimate the texture of how the 3D shape should be colored over time.
And what we found was that by enforcing, even though we don't have the label, we don't have the 3D model over time, And in addition, we have this object, which is a bird, which can deform non-rigidly, right?
It can spread its wings out. It's not a fixed 3D object.
But we found that by enforcing consistency of the parts of the bird and their shape over time, their texture over time, we can sort of enforce stronger constraints and better learning the 3D shape. by exploiting the temporal domain, which was really fun.
The neat thing about this is that this work is really relevant if you want to try to reconstruct, let's say, 3D animals in the wild, right?
We don't have any 3D, you know, we don't have 3D models of animals in the wild, zebras or or lines or whatever, all you would have probably is somebody taking really nice Zoom cameras and sitting far away from them in safaris and videotaping them, right?
So all you've got are these videos. Then the question is, if you, let's say, want to embed or create these 3D animals in computer graphic games or something, you need to construct 3d characters for that.
Right. And so that was a really fun project that, you know, How can you now take these in the wild deformable objects and try to construct the 3D shapes?
And so how do you judge success on something like that?
Are there somewhere reference, you know, reference 3D models that you can access or is it more of kind of an eye test?
Does it feel real? Yeah, that's a good point.
So yeah, there's a little bit of chicken and the egg there.
So we did two things. So one was... If your 3D reconstruction is right, you can then use those 3D reconstruction and its texture to create a 2D image. by using computer graphics, for example, right?
And then you can look at the silhouette of that reconstruction or the 2D projection and compare it to the input.
So that's one way of comparing it. That's one way that people do.
And similarly, you can look at like certain key points, like let's say the eyes should show up after the reconstruction and projection back to the 2D, the eyes should be where they were detected in the original 2D image and so on.
But there's another way is that we tested was like we had a few toy models, like we had toy zebras that we got. but they were rigid.
So we did some tests with rigid ones. Then there's a third way.
There's actually an interesting dataset that somebody collected at the University of Bath this year. where somebody actually took their dog into a lab and had them move. their pet dogs and they did use a Kinect 3D camera. to capture their motion.
But that's a really small data set. And I think it has like maybe less than 10 dogs.
But that was also a potential way that we considered, but then we didn't actually end up testing on that particular data set because we didn't actually reconstruct dogs.
But that's one another way. Yeah. Interesting.
And so the other paper. Yeah, so the other paper.
Yeah, that one's also really fun. So in the other paper, what we were trying to do was trying to take human faces and and manipulate them.
The particular two ways that we were trying to manipulate them was to move their eyes in the image such that you can change their gaze in a controlled manner.
So let's say you have You're given an image, the person's looking in a certain direction, and you say, hey, I want you to change the gaze by 10 degrees to the left or the right or the up or the down or whatever.
But the key is that you want to do it in a controllable manner.
And you want to be able to give it like a specific number saying, I want you to change it by this many degrees.
And the same thing for the head's orientation.
We wanted to be able to change the head's orientation.
And then the third thing we wanted to be able to do was that like we wanted to be able to control both the head and the gaze.
Separately, meaning if I tell you to change the head by a certain amount, you shouldn't change the gaze and vice versa.
Right. So in this particular work, what we also wanted to do was, let's say that you have a large data set of you know, people moving their head and their gaze, but you don't actually have labels for all of that data set.
So like you have maybe just a small part of it where you actually know the orientation of the gaze or the head in degrees. but the rest of it is basically unlabeled.
You just have videos, right? So that was the challenge that we were trying to solve is like, how do you, use data that's unlabeled.
This is where the self-learning part comes in. and do it in a controllable manner, disentangle the head and the gaze.
Actually, there was a fourth challenge that we were trying to solve, which was that, Oftentimes what happens is like if you have video data like this, right, it's not consistent over time.
So it can happen that, you know, suddenly while the recording is happening, somebody might come in and turn on a lamp.
Okay, right. And so a part of your video is going to have direct uniform lighting on the face, but then a second part of it may have slightly different lighting.
Mm-hmm. let's say people are watching their laptop and they're watching a movie and Oftentimes, as the content on the screen changes, the reflection or the light on their face is changing from blue to green to reddish, right?
And so you could have basically like the overall global lighting of your scene could be very different across the video.
And so you're also trying to disentangle that out of your video while you're controlling things like you don't, necessarily want when you change your gaze that your lighting also suddenly changes.
That's what we were trying to do. It turned out to be really fun.
We learned a way in which we could disentangle all of this extraneous factors, from gaze and head pose and all of them from each other and control them.
So this makes me think of what we were talking about up top with your work on automotive related systems and distracted driving and monitoring the driver, their gaze and what they're doing in the car. does this work potentially have applications for that kind of thing or are there different applications?
Yeah. It has applications for two different things.
One is to train systems that are analyzing gaze or head pose.
That's exactly right. So let's say I have a person and I only have like 10 labeled images of them with known gaze or head pose directions.
And then I just want to create like synthesized versions of their face in all the other angles. and be able to train a system specifically for them.
And that's an application that we actually show, which is called semi-supervised learning.
Like we can kind of create this fake label data Right.
For this person, we show that in the paper.
The other application for it is just more along the lines of content editing.
So for example, you could have a group photo taken of people and not everybody is looking at the camera.
Right. And so you can go and post process and make everybody look frontal.
That's called gaze correction. Oh, man, there's not to throw anybody in my household under the bus here, but I see this application for like the grumpy teenager filter.
Exactly, or the baby filter, right? Or the baby filter, right, yeah, yeah.
So you can make your baby's eyes turn towards the camera.
Right, right, right. We're speaking today with Shalini DeMello.
Shalini is a principal research scientist here at NVIDIA, and we've been talking about two papers that she and her team presented at NeurIPS just about a month ago in December of 2020.
But let's go back a little further in time, if we can, Shalini, and talk about your background.
How did you get into AI and deep learning?
What inspired you to get into the field?
And was this something from childhood or later on in life?
Tell us. Kind of a brief version of your journey to this point.
Sure, yeah. Thanks for asking, Noah. So I think...
Ever since I was little, I always enjoyed science.
So physics was my most favorite subject in high school.
And so I went to college and I got a degree in electronics and electrical communication there.
And I thought that electronics and electrical have some components of semiconductor physics and so on.
And so that sort of inspired me. to pick electrical engineering.
But I think along the way, as I was going through my undergraduate courses, I discovered signal processing. and I discovered that signal processing has a very nice theoretical foundation, I also discovered that signal processing is closely related to image processing, which is a means to manipulate and change images.
And I think I sort of discovered that I would say probably in my junior year in college.
And I just sort of, fell in love with the whole field of image processing.
I think besides physics and science, I also liked art and This sort of married both of them really perfectly because everything that I was doing with manipulating signals, I could immediately see their effect. on images and visual content.
And it was founded in really nice theoretical foundations, mathematical foundations.
And so it just sort of tied everything together.
And also, I sort of did a few projects, like I did a bunch of summer internships, but I was introduced to faith recognition.
And sort of this idea of pattern analysis.
And yeah, that was really fascinating. I mean, it's a hard problem.
We don't... fully understand cognitively how people do it. trying to solve how machines can solve it is really interesting and fascinating.
So yeah, that's pretty much it. I think I would say those internships really introduced me That was it.
I knew that in grad school, this is what I wanted to do.
I was really focused by the end of my undergraduate studies.
I applied to schools. for graduate studies that specialized in image processing, signal processing, and computer vision.
That has been my passion throughout. So you were doing your undergrad work kind of in around 2000?
Correct, yes. I went... to undergraduate college between 1998 and 2002.
Okay, so when did terms like computer vision and deep learning kind of first enter your vocabulary?
So I think deep learning entered my vocabulary much later when I was actually working at NVIDIA.
Yeah, okay. But I did, so I started off as, learning image processing and signal processing to computer vision, which this term I was certainly introduced to in my undergrad.
Back then, okay, yeah. When I first entered grad school at UT Austin in the first year, I took a course in neural networks, like a graduate class.
And that was my first introduction to specifically using neural networks.
And back then, it wasn't necessarily convolutional neural networks, which is what you would typically term as deep networks.
They were fully connected networks, but without the convolutional part.
But I actually used neural networks in the first research paper that I wrote at grad school.
Oh, cool. It's interesting having... these conversations and hearing the backstories of people like yourself.
And, and there's a through line that's clear, right?
From audio to, you know, signal processing to visual signal processing and your interests and then the through line to where you are now with what we call AI and deep learning.
But it's always just kind of interesting to ask when the terms came in to be into your own world or kind of how that through line works. progressed relative to the industry and these notions of things like deep neural networks and deep learning and all of that?
Right. Yeah. And so I think when I was in grad school, the general philosophy at the time with regards to neural networks was. they were somewhat regarded as a black box.
So most of my graduate work was focused around pattern recognition. the general thought process at the time seemed to be, which was, I would say, from 2002 to 2008 in that era, was that Neural networks are black boxes.
You don't understand why they come up with the solution that they do, right?
And they were clearly not at that point, the convolutional neural networks or AlexNet had not happened to really show that they can do so much better than hand-designed features.
The appeal of the hand-designed features was that they at least tell you what you're doing.
They're explaining to you in some sense how they're coming to their conclusion.
And so I think it was not until like 2012 when Alex Krzyzewski really showed that, Sure, we don't understand what they're doing fully, but they can do way better than what hand-designed features can.
And so that was really the turning point that convinced everybody and said, well, okay, fine, we agree.
So to jump back into the present here, as I said, we're taping this at the beginning of 2021.
So what's on the agenda for this calendar year for you?
Are you continuing the work that you presented at NeurIPS onto something new?
What are you looking forward to? Yeah, so there's so many exciting things on the horizon that I'm really enthusiastic about.
I'm really looking forward to continuing my thread of exploring how can we make AI systems learn with less supervision?
There's a lot of debate about whether that is really necessary or not.
But when I think about it intuitively...
I feel like as humans, we don't really learn quite in the same way. that AI systems do with so much label data.
And yet we're able to generalize and yet we are able to you know, learn new concepts quickly and so on and so forth, right?
This to me as a researcher constantly puzzles me.
We are as humans able to learn effectively with less data, with less supervision.
And so how... can we make learning systems do the same I think this is a fundamental question to answer for the viability of AI and its future.
I keep thinking at the risk of oversimplifying things or showing my own ignorance.
I mean, is this sort of meta-learning and unsupervised learning, is this kind of a a step towards general AI.
It is, it is people, but then again, this is a very moot point, right?
It's very hotly debated. what is general AI, right?
Like what its definition is. But I wouldn't go so far as to say that is a step towards general AI, but I think for me, it's more of a step towards making it more ubiquitous and available, and also a step towards perhaps moving away from brittle systems that are highly dependent on the data.
I also think about a lot of systems, there's the question of ethics and fairness, right?
Like when your data sets are biased and it is the case that if you're sitting in one geographical region, you are going to introduce some bias in your data.
Absolutely. Right. Like it's, it's, inevitable to have that bias.
If you have a system that is not solely learning from what it has seen, but is maybe understanding the underlying compositional concepts better, it may better be able to generalize to new data that it hasn't seen, right?
And so for me, it's like a question of cost and viability. of AI systems developing further.
It's also a matter of fairness and enabling other societies which may not actually have the ability to collect so much data and annotate it, be able to also benefit from what AI has given to us.
The whole time I was listening to you, I was trying to form some sort of a joke that wouldn't fall flat along the lines of, Well, give us a call back when you start working on complicated problems because really this is –
But the whole time there's this through line I'm hearing of you're just drawn to these complex problems and then finding... finding out of that the applications that will both progress the field, but also with a strong humanist almost sort of moral stance, right?
A human first moral stance to it, which is so necessary the more we rely on this stuff.
So kind of a final question, Shalini, looking forward, what advice would you give to the generation of researchers, either folks that you maybe have worked with or even kind of thinking further ahead to The people who are kind of just falling in love with with physics, as you did, or computer vision or signal processing or whatever it may be.
Any advice that you could give to them as they start to embark on their studies and careers?
Yeah, sure. That's a great question, Noah.
I think what I would like to say is that I think looking back at my career, the one thing that I have learned is that it's really important to follow your passion, right?
There's always sort of the more lucrative choices. that you don't feel as passionately about.
But at the end of the day, what will keep you really motivated or what keeps me really motivated is to work on things and problems that I feel deeply passionately about and my core values are aligned with.
And I think the other advice that I would give is to always be curious.
I think that that always having the genuine curiosity like a child and questioning everything and trying to dig deeper into every little concept. really pays off and thinking about problems deeply.
And it leads to you becoming a better researcher, a better scientist. rather than having sort of just a superficial understanding of things.
So yeah, be very curious and follow your passions.
Amen to that sage advice. For folks listening who want to keep tabs on your work, do you have a research blog that you publish to or where can folks go online to find out more about what you're doing?
Sure, yeah. So off of research.nvidia.com, there's a link to all the AI researchers at NVIDIA, and I have a personal page.
If you search under people, you can find all my publications there.
Great. That's a good landing page. And the NeurIPS papers we referenced, those are up there too?
That's right. Yes. Great. Excellent. Well, Shalini, thanks for taking the time to come on.
I wish we had more time to get deeper into some of these things.
But this will have to start us off and maybe we can catch up again down the line and see how you fared in 2021.
Thank you so much for taking the time to chat with me about my research.
I would love to hear from you again. Absolutely.
The pleasure is mine and the listeners, I'm sure.
And all the best to you and your team with your research and, of course, staying safe and healthy this year.
Thank you. You too. ¶¶ Thank you.
