Welcome to NVIDIA's AI podcast. We think of machines, especially our computation powered smart machines, as objective.
They perform assigned tasks endlessly with the same precision and lack of emotion.
It all comes down to ones and zeros after all.
But our guest, Chris Hammond, he's the chief scientist and co-founder of Narrative Science and a professor of computer science at Northwestern University.
He points out that there's lots of bias in our so-called smart machines, and they come from different sources.
And that's what we're going to talk about, bias and how it creeps into our systems.
Chris, welcome. Thank you for having me.
Thank you for having me. I'm looking forward to it.
Chris, you did this great story for TechCrunch, and you pointed out that we have bias in all kinds of systems.
And maybe you can tell us, let's walk through kind of the types, but how is it that we misconstrue what our systems are all about to begin with?
Well, I think that we would like them to be unbiased.
In fact, there's a very strong sense in which we would like the artificial intelligence systems that we build To be sort of it is cold, emotionless.
So we can oddly enough, so we can make fun of them. because they're not as clever and good and creative as we are.
But the reality is, is that we build them.
We train them. We sometimes give them the reasoning rules they're going to use.
There's absolutely no way we can avoid letting all of our notions of how the world works creep into these systems.
Because we are the ones collecting and gathering the data that they train on, because we send them in a certain direction to solve a certain problem, we can't help but let our own biases creep in. into what they do.
The one way to think of them is, if you think of AI systems, and unfortunately, that's the way I think of them, If you think of AI systems like, you know, like children and just as our biases. creep into how we talk to, we train, we teach our children. they creep into the way in which we talk to, train, and teach our AI systems.
Let's walk through some of the examples.
I know you have several different categories of them.
Let's talk through some of those. We mentioned how we train... smart systems on data, and so there is data-driven bias.
How does that crop up and then, and I get it.
I'm biased in the sense that I learned how to speak English first.
And so that's sort of a data-driven bias because that's what I was exposed to and that's what I was trained on.
But what are some examples in the AI world?
Well, I think one of the more benign, though interesting examples that cropped up early on was There was a team that decided they wanted to teach a visual recognition system, a face recognition system. to recognize beautiful women.
And so they gave it a whole bunch of examples of beautiful women and it learned how to recognize, quote, beautiful women.
But of course, it was the team's notion of beautiful women.
And it turns out the team was... you know, a standard set of, you know, relatively young white guys.
And so the system itself would recognize as beautiful their notion, which was of attractive Caucasian women. who looked like they were born and bred in the United States.
And when you look at that, you realize, oh, that's fine.
But don't call it something that it's not.
That is, it didn't learn anything. It didn't learn the notion of beauty.
What it learned was these individuals' notion of beauty.
And so if you sort of step through what kinds of example sets – systems are being trained on, then you have to pull back and go, where does this example set come from?
Are these examples are these examples that are really associated with some sort of ground truth?
Or are they examples that this collection of people selected?
And you could do this. You pull back and you look at that and go, oh, imagine I want to train a system on. you know, recognizing the credit worthiness of individuals.
Well, then I give it examples. And those examples might come from some objective source or they might come from the fact that I've already got examples from positives and negatives that were positives selected by somebody else.
And so it's skewed already in a particular direction.
And there's a moment where you look at that and go, well, that's not avoidable.
And whether or not it's avoidable is something we can contest.
But you have to understand that for any given system— for any given system that is trained on a data set that you hand it, the first question you have to ask is where does that data set come from?
Is it already skewed in a particular direction?
We just talked about beauty and credit worthiness.
Something like beauty almost doesn't seem like something that data can even encompass.
And so is it also a question of what you're trying to do? train it on at some level.
So you're saying beauty is in the eye of the beholder?
Is that what you're saying? I guess I am.
Credit worthiness seems like a much more, maybe not binary, but calculatable, right?
As opposed to something as... as intangible and kind of subtle and changeable as beauty?
Maybe yes, maybe no. I would say that for a lot of people, they would look at you and they would think this is subjective.
I mean, the reality is that it might be subjective, but it's If you look at large populations of people who come from the same cultural background, their notions are really converge pretty quickly.
But for credit worthiness, it's like, well, It depends upon where you want to draw the line in creditworthiness, how much risk you want to take on. compared to the benefit that you might have.
So I actually might be evaluating credit worthiness for a particular possibility based simply upon the odds that someone is going to pay me back.
But it also might be that even though it might be less likely, ever so slightly less likely that they'll pay me back.
That if they do pay me back, I will then have a customer for life.
And so it could be that you want to have the cost of doing business in there.
And that is not, I mean, that's not immediately apparent for most data sets in terms of things, decisions that we have made in the past.
And there's also the possibility that because of the decisions that we made as to yay or nay to begin with, we've actually skewed the outcomes.
And the very act of turning a class of people down for credit might actually push them in the direction of being less credit worthy because in fact they are looking at more and more riskier scenarios with regard to their credit or higher interest rates with regard to their credit.
There are a whole bunch of features there that skew you in one direction or another that might not be explicitly mentioned in the data, or in the characterization of what it is you're learning.
And the frightening conclusion, I guess, in some ways, or the disturbing conclusion could be that it's sort of self-fulfilling, too, that their credit worthiness then therefore drops even lower because of your bias.
That's interesting. Oh, absolutely. Because credit assessment is not usually in the binary.
It actually has rates usually associated with it.
And it could very well be that you would do a great job of paying back your loan if you have a rate of 1%.
But you might have a struggle if you have a rate of five.
And because in the past, People like you have been pushed into the 5% category You look less and less credit worthy, which means you are even more pushed into the 5% category as I build a learning system around you.
Another bias that you describe is bias through interaction.
And you have this great and sad story of Microsoft's Tay.
Walk us through what happened there and how this is something to consider as you are building a system.
Well, I mean, for some for some systems that are learning systems, they don't learn by, you know, they don't learn from giving it being given a batch. of data, an entire data set, and I'm told to learn from that.
They learn over time by interacting with multiple users.
Um, and Tay was a, a call and response system that is you would, uh, you would, you would, uh, In some sense, ask it questions.
It would give you answers. You would tell it things.
It would remember those things that you told it and use that as Christopher the Milfer. for follow-up, for answers or comments that would make me making to other people.
When it was released in the world, there was a group of people who decided, let's have some fun here.
And what they did was they they started making comments today that tended to be misogynistic, bigoted, racist. just fantastically. objectionable.
And Tay learned these things. But Tay learned these things in much the same way that a parrot would learn these things.
I mean, you could have an unpleasant person who trains a parrot to be incredibly obscene.
And it's pretty easy. And with Tay, it was a concerted effort.
I mean, it was really a group of people who decided they wanted it to be this way.
And so it very quickly converged on responding to the terms that it would see with objectionable comments.
And within 24 hours, it learned this. The thing that's amazing is that Tay has a counterpart, and Microsoft just took it down.
And a lot of people see this as, oh, this is like a failure of AI.
And it's like, no, it's actually a failure of of us as a species.
It's so unpleasant. Apparently the people at Microsoft have not spent much time on Reddit, is all I can say about that.
Oh, no, that was the thing. Actually, the thing that killed me was that Microsoft, I think Microsoft very cleverly had not just developers working on this, but they had some comedians and improvisers.
And there is... There are very few comedians, or in particular, improvisers, and this is something I care about because I live in Chicago, And we've got Second City in Chicago.
But there are very few improvisers who don't who have not had the experience of you ask for a suggestion from the audience.
And people come back with, yeah, believably, you know, it's usually just blue, just just, you know, see things.
Something about your mother. Yeah, that's exactly right.
And and it's like, did you not see that coming?
But I think part of it is, though, that Tei has a counterpart in China that's almost exactly the same piece of code. and it's flourished there And it's a very pleasant system to talk to.
And it's not at all like what Tay became.
And it's because from a cultural perspective, there was nobody there who wanted to skew it. and train it to be a bad thing.
And so it's not a bad thing. And this is actually when we start thinking about conversational agents and learning based upon interactions.
We have to start thinking about, well, what does it what are going to be the constraints that we put on that learning so we don't bias those interactions in the direction of those comments that the genuinely objectionable might want it to give.
The thing for me, though, is that Tay is – Tay really is like a parrot.
And I tried to explain this to somebody once.
It's like – If Hitler had a parrot and it was objectionable, you would not call Hitler's parrot a racist.
It's just a parrot. If Hitler's barber... said the same things.
You would say, oh, Hitler's barber's a racist, because it's something that is actually listening, understanding and then responding.
But Tay did not really understand. It was really a call and response system that was just saying, oh, this is something, given that you just mentioned Ricky Gervais, someone else mentioned Ricky Gervais and told me and told me something about his sexual orientation.
I'm gonna come back to you with a comment about his sexual orientation. using the vilest language, the vile language I was given.
It doesn't, but it doesn't know who Ricky Gervais is or who or what the language means.
It just knows that there are these words that are collected together from a statistical point of view. that it can spit back at you.
That grows through interaction. Actually, if there are contravening forces, something that pushes in the other direction.
You can correct for that, but it really was that group of people decided this was going to happen.
And there was nobody on the other side deciding it wasn't.
Yeah. And so in that case, does it need human oversight then at that level or at that point i mean to to correct it and get it headed in the right direction and therefore okay now you can go out there on your own and and we trust you not to turn into a you know, a horrible person or system?
Or are there other ways to account for our own cultural massive speed bumps, as it were?
I mean, if you're thinking in terms of a genuine AI system, the way you correct for that is that you have you have the system actually understand what it's being told and what it's saying.
And and so, I mean, and for me, I mean, for me, I look at this and it's like this is not.
This was not a failure of AI. This is a failure of pattern recognition and the use of statistics with no semantics.
And if the system actually knew something about what it was saying, it wouldn't say those things unless you told it to – unless it wanted to say those things.
And that's the way you correct for that.
You can't correct for it at the pure... at just the term level because then what you get is people who are and this is People who will find more and more creative ways to say things that are just as objectionable.
If you take a look at – there were a lot of – certain game environments for younger children.
And they have some very, their rules around what you're allowed to say and what you're not allowed to say.
And you've got kids who are tossed off of them because they say they use obscenities.
And then you look at the tracking, and unless there's a human who's actually looking at what people are saying, you've got kids who are saying more and more objectionable things. finding very subtle and oblique ways of doing so.
So they teach kids how to be very metaphorical, which I think I guess is great.
They're metaphorical in an unpleasant environment.
You are also an adjunct professor in journalism, and this whole question of bias, certainly we—
And I'm laughing in a sad way. We saw that front and center lately, whether it was fake news or getting. only the news that we like or we think we like, and I mean that both literally and figuratively.
How does bias creep into our information and how are we participating in that as readers and watchers and likers and you know, sharers.
It's going to sound strange to talk about it as bias, but one of the things that has been deadly in terms of, I think, news. is the notion of similarity.
So, you know, you go to, when you go to, when you work on a site, a community site that's like, you know, a social community site like Facebook, that you get pieces of news that come to you from people which is great, you're friends.
But anything that else that is suggested to you can come from can come from sources where an analysis is done of the things that you look at and click on, and you get things that are similar to them.
Right. And this shows up everywhere now.
It's sort of the way that you get presented things on Facebook.
You go to Google News, you get similar. The notion of similarity works tremendously well in search.
So when I go and I go to Google or I go to Bing and I type in Florida vacation and What I want is I want documents that are similar to those words or include those words.
And so I want Florida vacation documents.
I don't want something coming back to me that says, Oh, maybe you should not spend all your money on vacations.
Maybe you shouldn't go to Florida. Because you'd look at that and go, why?
But that doesn't help me. I want things that are similar.
And when we look at Facebook and we look at sort of news sites, Um, what we're getting, what we often get back is things that are similar and in fact you will, you will even, they'll even be named as such.
So things that are relevant to what you've just said or similar to what what you've just been looking at.
But if you think about an op-ed page for a moment, If you went to a newspaper and the op-ed page had one op-ed, had one piece of opinion, and then a set of other opinions that were exactly the same.
You would look at them and go, oh, there's been a failure of editorial judgment here.
And so in the news space, you want point counterpoint.
You want, or at least in the opinion space, point, counterpoint.
You want something from one point of view, something from another point of view.
But in fact, the technology to do that is hard.
The technology for similarity is easy. So if I'm looking at a story about a right to life, It's more difficult to get to stories about a right to choose. than it is to get other right to life stories.
And so from a technical point of view, If I'm an engineer in particular, I might actually skew you in that direction.
And so what happens is that more and more we see a world where it is, the bias is actually ours.
But it's a bias because the technology is not able to move us away from the point of view we already have.
And so we have a reinforcement of what tends to be called, in terms of humans, confirmation bias.
We only look at the things that actually confirm our ideas.
But now we have technology that actually reflects that.
And at some point, there is a moment where we decide. we actually can't do this anymore.
That whenever we from an editorial, from a news perspective, There are different things at play here above and beyond similarity. similarity is not enough and focus on developing the technology suite that will get us to this notion of point-counterpoint. difference, the arguments against when we're looking at arguments for.
Do you hear those kinds of discussions happening or read those kinds of discussions happening and think that people are heading toward a solution or will we revert back to... our own similarity biased kind of ways.
I don't see a massive amount of conversation right now.
I mean, I think there is. the absolutely reasonable focus on fake news.
But fake news is nothing. I've got to tell you, fake news is nothing compared to real news when you can't escape your filter bubble.
Fake news is just another. The fake news you get. is fake news that is similar to the news you are reading.
And the reason why people actually embrace moments where they will read a piece of fake news is it confirms what they already believe or amplifies what they already believe.
And it wouldn't work. if it didn't do so.
I mean, if I'm reading political news and I see a story about how giant clams are taking over our political system.
I would look at that and go, well, that's fake.
That's something that's ever so slightly close to what I believe. that, you know, that the, you know, that the election is rigged That the people I believe in are doing good things.
The people I don't believe in are doing bad things.
Those are all just like, they just accrete along with the other things that are existing in my bubble.
Right. I mean, I love this idea of I'm searching for a vacation to Florida.
And what I get in response is one of the responses is. well, maybe you shouldn't be spending the money on a vacation.
But in the sense of if you're looking up a word, say, and you're looking for synonyms, At the end, there's typically an antonym too, and I think that maybe our results need that antonym at the bottom too. on occasion just to remind ourselves of, oh, wait, right, maybe I should think about this differently.
I think you are absolutely spot on. And unfortunately, on a word level, it's easy to get an antonym.
But on an idea level, it's really hard. Because, again, just as the reason why Tay had its problems is because Tay didn't really understand.
And the reason why it's hard to get conflicting points of view or different points of view based upon a document. is it's hard to look at that document and think, what is this about?
What is being said here? And I can't just look for all the words with not in front of them, because in fact, we're clever enough to understand that that the opposite of right to life is not no rights to life, it's right to choose.
And it's like choose and life are not opposites.
So it's very hard to get to those documents.
But in fact, I think that is what is, if we want to have a genuinely informed citizenry, which I believe is the goal of the entire journalism, you know, the entire field of journalism. is we need to actually think in terms of what are the technologies that are required in order to get us there.
And technologies that can find the opposite are part of it.
There's a final kind of bias that you describe that sounds like it's related to our similarity bias, but that it reinforces stereotypes.
And you, call it conflicting goals bias.
It's really, it's just the notion of, I might build a job site.
And my I make money by having you click on, you know, having you click on jobs that you might want. that you're gonna apply to.
I mean, that's how I get money from the people who post these jobs.
Now, so my goal is to have you click on as many things as possible and apply for as many jobs as possible, which sounds like a great goal.
But if it turns out that if it turns out that I give you jobs that are jobs that are great jobs for you.
But you don't have it in your head already that these are great jobs for you. you won't click on them.
And so what I, for me, and I won't even think about this explicitly, but for me, if I had a learning system even that helped me do this, I'm going to suggest the jobs that are the most likely ones that you're going to click on.
None of the best ones for you or the ones that are the best matches for your. for your skill set.
So I will tend to skew in the direction of certainly jobs that are gender appropriate.
So I will give you if I'm a female who is looking for a job in the medical world and there are jobs for medical technicians and jobs for nurses.
I will get more jobs for nurses presented because I will tend to click on those more.
Now, it turns out there's a human piece of this which is particularly pernicious, and that is If you are about to enter into a task, And there was a stereotype associated with you or whatever your demographic is and that task.
And I remind you of that stereotype before you start doing that task. you will fall into that stereotype.
Nerds being reminded they are clumsy. will not do as good a job putting things together or doing physical activity.
And so I'm not reminding them. Women reminded that women aren't good at math, even though that has nothing to do with reality.
We'll do worse at math problems if they're reminded of that beforehand.
And the problem here is that being presented with a set of jobs that actually match up with your stereotype, reminds you of that stereotype, And you're thinking in terms of that stereotype, and you will click in the direction of that stereotype.
Now, this is another place where no one is doing anything wrong.
No one's doing anything wrong. The company that is putting out these putting out the jobs, they just want trying to make money and they want you to get they want you to get hired.
Then when you click on things, the people who are trying to hire you, they just want you to get hired.
You want to get hired. All these things are good.
It's just that they come together. to really absolutely unintentionally create an environment in which these biases are reinforced.
And this is actually one of those areas where Being able, understanding what it means for human beings to make decisions, to be presented with options. and then have systems that might be learning in association with those options and decisions. is really necessary if we're going to sort of push things forward.
Do you think, though— stepping back that this is all human bias as you describe it.
These are all in some ways kind of our problems.
Can we expect our machines to solve our problems for us if we can't solve them ourselves, if we hold these stereotypes so closely, if we have these towards similarity and, hey, I just want to read what I know or what I like.
How can machines help us break our own bad habits or our own really bad as you say, pernicious habits.
We design them to do so. I mean, absolutely outside this realm, and that is the braking system on your car.
I'm looking out the window and I've got ice and snow everywhere.
And if I'm driving my car and I start skidding, In the old days, if I slammed on my brakes, my brakes would seize up and I'd go into a skid.
Nowadays, I slam on my brakes and my car says, oh, OK, I get it.
You want to stop. By the way, the wheels have stopped and we're still moving.
I'm going to actually... break and unbreak, break and unbreak really quickly so that you don't lose control.
And so the car itself is actually correcting for a problem I have, a mistake I've made.
And I think and it's not because it's so smart.
It's because the people who designed it were really smart.
And I think that we're now entering a world in which as more and more intelligence comes into play in our systems, that we now have to think in terms of how can we build systems that will help reinforce the best behaviors.
How can we build systems that teach us about best practices?
How can we build systems that actually help us reason better?
Because we human beings, God bless us, we're a collection of, like, vaguely serviceable heuristics and a complete misunderstanding of statistics.
And having machines help us. because there are people who understand who we are, how we are, and how we think. and actually designed those machines to really cater to the best of us.
That actually, I think that's absolutely doable.
So it's not the machines doing it. It's us doing it, but we're doing it at scale.
And to do so, we need the machines in the middle.
It's exciting. I mean, it's exciting. Like you say that it's doable and that we actually have a sense of ourselves enough and a sense of the things that we're building. to take a crack at it.
I do think, I'm sorry, I do think that, you know, it is when you do that search for Florida vacation.
And it's not so much you shouldn't go on vacation, but having a sister come back and say, have you considered meditation?
If you meditate during the week, during the year, you can get yourself calm and collected all the time.
It would be fantastic. Where it's like, I understand where you're going.
I understand what you want. I understand what you need.
And I will help you. And I will help you because I was built by people who are trying very hard to help you.
And that's the future I'm hoping for. Chris Hammond, I will see you in meditation class, not Florida.
Thank you so much. Oh, no, thank you. I appreciate that, Michael.
This was great. Postscript here. If you are listening to this podcast on iTunes, Google Play Music, or whatever your favorite podcast player is, do us a favor and leave us a review.
Thanks for listening.
