Welcome to NVIDIA's AI podcast where we explore the expanding world of artificial intelligence.
It's been 20 years now since IBM's chess-playing computer Deep Blue beat world chess champion Garry Kasparov in a six-game rematch.
More recently, Google's AI-powered AlphaGo beat Lee Sedol in a five-game Go match, and that was just last year in 2016.
So it turns out we love building machines and systems to challenge humans and games.
And that is what our guest, Professor Michael Bolling, has done. but not for chess or go, but for poker.
And to be precise, heads up, no limit, Texas Hold'em.
Professor, welcome. Thank you. I'm glad to be on the show.
Well, Professor, can I call you Michael?
Yeah, please do. All right. You're a computer science professor at the University of Alberta, the leader of the computer poker research group. and one of the creators of what you guys call Deep Stack, an AI-powered... poker champion that recently beat a field of professional poker players.
For those of us who aren't professional poker players, What is Heads Up No Limit Texas Hold'em?
As it compares to games like chess or go, how is it different as a plot problem for the likes of you and your computer science folks to solve?
Poker is a vastly different game than chess or checkers or Go.
These sort of board games where where the primary description of those games is that the board encompasses the entire state of the game.
And so if I want to know what's going on in the game, I just simply look at where the pieces are on the board.
I don't even have to remember how I got there. because that's not important anymore.
With a game like poker, it's the sort of quintessential game where that's not true. that there's in fact a piece of information about the game, the other players' cards, that I don't know.
And that is fundamentally what the game is about.
If you were to try to play this game where all the cards were face up so everything was knowable, this would be really one of the stupidest games you could play.
Right. I can see that. The excitement comes from that.
The hard and the AI challenge comes from that, that I don't know what cards you're holding, But it's not just that I don't know.
You start to tell me. And you start to tell me by taking betting actions.
So the way a game of poker works is that we're dealt some cards and we're all going to sort of bet who has the better hand.
New cards will be revealed, maybe added to our hand or flipped over on the center of the table to change the value of our hands as we're playing.
So, as we're making bets about who has the better hand, the strength of our hand is also changing.
But here's the thing about poker, and this is what I find fascinating, is that, you know, I imagine, you know, wearing sunglasses and a hat, but that, you know, when we watch all these professional poker players play, there's this, you know, kind of stony-faced looks and my impression is that it's all about the human aspect at a very high level of me knowing kind of What you're doing based on the twitch of your eye and the flick of your tail in your hand.
Yeah, so we often take a problem like this one where I don't know what cards you're holding, and we try to turn them into these very human-like problems where it's about... looking in your eyes and telling if you're lying or not.
One of the things that's changed in the last 20 years of poker was really online poker.
And so the game actually was starting to be played over the internet with computers, no cameras looking anywhere, and no tells that could be had in that way.
It turns out the game is still both fascinating, and hard to play, and that there are professional players that can win exorbitant amounts of money against much worse players in this department.
And they're not doing it by looking in your eyes.
They're doing it because you give away your cards based on how you play.
Another factor is that if you're playing at a high level, if you're playing two high-level players are playing against each other, If they have some obvious tell, like in the movie Rounders with how you eat your Oreo cookie.
If you had that, you're not going to get very far in the poker world.
So largely speaking, even under live play, a lot of that sort of aspect of the game, you're not going to get very far. far if you're giving your cards away.
So, you know, in some sense, you know, the humans are playing as if they were poker playing machines. trying to remove as much of those human elements as they can in terms of giving away what's going on in their hand.
That's right. It's clearly an important part of your own play to not give away your cards, especially not with sort of how you hold your cards or where you put your chips.
But you also don't want to give away your cards with how you play.
If you only bet when you have good cards... then you're effectively telling your opponent exactly what you're holding based on your bets.
And you want to avoid doing that because I don't want to say this, one of the good places to be in is when your opponent isn't sure about what hand you're holding.
And it's valuable to be in those kinds of positions, so you want to preserve them as long as you can while still, of course, making the pot big when you have good cards and trying not to lose too much money when you don't have good cards.
So how do you, whether you're a AI system or whether you're a human, win at Texas Hold'em?
How do you train a machine to win along those same lines?
Poker, one of its fascinations for a long time was it wasn't obvious how a human player would build up good expertise at this game.
I think what you saw a lot of people, the answer for a long time, 10 years ago was just play a lot of poker hands.
And then this – whatever is going on with the human brain that is able to internalize with just tons and tons of experience can internalize that into – some, you know, maybe even subconscious knowledge of how to play, that seemed to be what you would do.
Over time, and computers already had an impact on this even 10 years ago, there started to become a notion that, hang on a minute, this game can be analyzed mathematically.
I guess scientifically we knew this from game theory going back 50 years.
But it started to become possible that you could think about the game much more mathematically and come to real answers about the right way to play and with what probability should you actually bluff versus just folding when you don't have good cards.
Those actually have mathematical answers to them and simplified settings.
If you can start to internalize the simplified settings, maybe you could figure out how to apply them in the more complex ones.
And so that started to change how people played the game.
There started to become a whole wave of players that were game theory wonks and were studying how to understand these small little toy games where you could analyze them outright. and using that in their way of playing.
More recently, we've started to see computers used to solve simple poker situations, much like you see in chess. where computers will come up with end-game answers, and humans will study these situations that the computers are deriving the correct answer for,
And they'll try to internalize why is that the right answer.
But what you described that where computers kind of anticipate the end game and then they make a move depending, you know, based on that is this kind of serial approach to the game where, uh, You've got some cards that I can see.
I've got the cards that I can see. And I anticipate every single possibility. going to the end result, right?
But does that work for a game like heads up, no limit, Texas Hold'em?
No, I mean, that is in fact the crux of why that game is interesting and why we're super excited to build DeepStack.
Up until this new program, the typical way that you would approach a hard problem like poker from an AI point of view is you would do the exact opposite of what you would do if you were building AI for chess.
So AI for Chess, or Go, does most of its thoughtful computation while it's playing.
There's often a ton of computation sitting there while it's playing to reason through how to play out this exact position.
That wasn't how we approached poker problems before this, including how we beat pro players at heads-up limit 10 years ago.
So notice I said heads-up limit, not no limit.
The heads-up limit game is far, far smaller.
Anytime you make a bet, it's of a fixed size that's predetermined.
And this drastically reduces the possibilities for how the game could be played out.
And that made it possible... That you could reason about all these situations ahead of time before you even arrive at the particular poker game you're playing.
And that's what programs would do. They would think through all the different ways the game could play out.
They'd figure out a play in every one of those situations in a way that would ideally guarantee that no matter how the opponent plays. that it wouldn't lose any money.
It was trying to strive to be the perfect poker player, one that couldn't lose.
Now, if the opponent got dealt better cards than you, then yes, you could still lose.
So this is a statement about on average, if you played enough hands that the luck evened out.
Right. And I was going to ask you about luck, because I guess even computers can get lucky, too, like just because you're a computer doesn't mean you can't get lucky.
Oh, absolutely. It's a huge part of the game.
And in fact, it's one of the challenges to our claim of when we built DeepStack.
One of the challenges is how do you How do you even show you're better than poker pros when it can take upwards of 100,000 hands to really separate? say, a pro from an amateur player just by looking at the money exchanging hands on each hand.
That's how much luck there is in this game.
And the only reason pro players can sort of make a living doing this is because they often play as much as a million hands a year.
And so they just have to be playing lots of poker to overcome the luck.
But that makes it harder on our computer programs to find or really to get that much time from a human player to really do that kind of analysis.
And describe how the hands play out. It's two cards that are dealt down to each player.
You're playing against one person for starters, correct?
Correct. You're just going to play against one opponent in the heads-up game.
You're dealt two cards face down, and so are they, and there's going to be a round of betting.
And the way around a betting works is it starts with one player actually, um, For the very first round of betting, someone's already been forced to make a bet, and someone's been forced to make a raise of that bet.
These are called the blind bets. But then after that, one player is going to be facing a bet, one of these blind bets, and they could call that bet, meaning they just match the amount of money.
Or they could fold their hand and they just give up what they've already put into the pot.
Or they could raise the amount of money.
Now, each side has a certain number of chips already in front of them.
And so when you raise, you can raise any number of the chips you still have in front of you.
If you raise now, the other person's in the same place you were in.
They could call that bet or they could fold and give up what they've already put in. or they could re-raise on top of that.
But eventually it's going to happen that it comes down to a call, and then some cards will be revealed.
This is the Texas Hold'em part of the game.
And some cards will be flipped up in the center of the table.
And these are called community cards. They get shared between the players.
So you're trying to make the best hand you can. from the shared cards in the center of the table, plus your two cards that the other player doesn't know. round of betting another card will be revealed there'll be a round of betting and then a final card will be revealed with a final round of betting If it gets called all the way to the end or both sides end up all in, meaning all their chips are committed, then you'll basically flip your cards over and you're going to form the best five card poker hand you can from your available cards.
And whoever has the best hand then wins all the chips that were in the pot.
So why is it that these multiple rounds of betting makes that game so much more complex and so much of a more difficult problem for you guys to solve? there actually are multiple rounds of betting in the limit game.
So, so it is true that the multiple rounds of betting does add complexity.
It adds complexity because the strength of your hand is constantly evolving and And so there are actually hands for which your probability of winning with this hand are similar, but they actually are very very different kinds of hands. and it's because this hand can't really get any better with new cards, but a different hand might be able to get better depending on what future cards land.
And so you have to think about there's all kinds of different types of hands.
The hands aren't just sort of lined up in a line as to which ones are better than others.
You could be only one card away from getting a straight, so that's five cards in a row.
And that card, if you don't get your straight, then that hand is worthless.
But if you get it, you're likely to win the hand.
And that's a very different kind of hand than having like a pair that you already have the pair.
So the multiple rounds means that the strength of your cards is constantly evolving as you're betting, and so that one bet on the previous round might not your cards might change for the next time you're about to bet on the next round.
And again, as a player, whether you're human or not, you're looking for signals sort of to to know what that other person's hand is playing out to be or becoming as that game goes on.
That's right. And if you can put them on a particular pair of cards, that's super powerful.
Because if a seven lands and you can become convinced there's no way they're holding a... you know a pair of sevens or even just one seven in their hand and you can be certain that this this card could not have helped them And that's a powerful advantage when you go into the betting realm.
Are you a big poker fan? Like you've been working on computer science and poker for, you know, 20 years now, right?
I love games. That is near and dear to my heart.
And I love thinking about how AI can approach games and thinking about how humans approach games.
It turns out that I'm not much of a poker player.
I actually don't have much patience for the game.
I'd be like five hands in and I would just like go, I'm going all in.
Let's see what happens. whereas poker takes a lot more patience and psychological control than I typically have.
And that's true of a lot of people in our group.
A lot of our group You might think that, yeah, that we're a bunch of degenerate gamblers, but we're actually—
AI researchers who see poker as being a truly fascinating problem, for many of the reasons you already described, we so often think of it as a psychological game. as about bluffing and determining what the other player knows and manipulating the other player's beliefs.
And so how do we get an AI system to have those seemingly human qualities in a way it plays the game?
Well, so how did you do that? Because I know bluffing is still a part of the game. as you say, force the opponent to do something that they probably shouldn't based on bad information.
How do you teach an AI system or train it on data to be a world-class professional poker player?
The idea is essentially to train a poker player by having it play against itself.
Like that's, that's at the heart of the premise.
And it turns out that if you do this in the right way, you can actually get the system to learn to do things like bluff.
And the reason it works is because as it's playing, it's going to be evaluating each of the actions it plays.
It took a particular action. It's thinking this is the right thing to do.
But it's going to be evaluating all the other things it could have done.
It's basically going to be computing how much regret it has for not having done that.
So suppose it's starting to learn to play poker.
It starts off by acting randomly because it doesn't know what else to do.
And so it just starts playing randomly against itself.
And over a while, it realizes... Well, when I have good cards, I probably shouldn't just randomly fold them.
It would be better off. I have some regret for folding those good cards.
I probably should have raised with them.
I would have won more money. And so it starts this tilted strategy to raising more.
Now, when it's in the other seat, it's playing against itself, so it's got to be a little schizophrenic here.
It's got to imagine, well, when I'm the other player, when the opponent raises...
Now they're starting to raise not when they have random cards.
They're starting to raise only really when they have good cards.
So I should probably fold them. Ah, so you're starting to figure out your opponent and their style of play and patterns really, yeah.
And now let's go back to the first player again.
Oh, my opponent actually folds when I raise.
They don't fold when I raise when I have good cards.
They actually will fold when I raise even when I have crappy cards.
So now I start to raise when I don't have good cards.
So this back and forth sparring actually starts to get the probabilities right with exactly with what probability it starts to learn with what probably should I raise So as to make you essentially maximally confused and not be able to take advantage of knowing what cards I hold.
And the opponent needs to, its counter strategy on the other side needs to think, with what probability should I be calling to make sure that you're not motivated to just raise me with any pair of cards. because you're gonna block me off my hand.
So how does that apply then? I understand how if I'm playing against myself, as it were, as an AI system, There's a consistency there, because I'm only playing against this one set of, I don't know, data and attributes.
But don't people have different styles of play?
Is what you're describing kind of reducing poker to, like you say, its statistical essence?
It's a little bit like that, and you can do this partially because we're looking at the two-player game.
In the two-player game, this is kind of like chess.
So if you look at how these chess programs work or Go programs work, They're doing very similar things.
They're basically playing against themselves, even in their reasoning for the move they're about to make.
They're sort of reasoning about, well, if I do this, this would be the optimal thing for my opponent to do.
But if you play against a particular opponent, they might not do that.
And then what happens next? Well the key factor in a game like Go is that if they did something else, it would be worse for them.
And that's a factor in poker as well, too.
It turns out that it's going to play in such a way where it's essentially reasoning, if my opponent were to optimally play against me, here's what I would want to do.
If they don't play optimally against me, that's only in my benefit.
So while I'm not going to explicitly exploit those errors, I'll just take advantage of them every time my opponent makes a mistake.
So 10 years ago, this was in the Heads Up Limit Texas Hold'em.
How long did it take to play a game, and how long did it take for the system to make a move?
So let me say, we actually... Only two years ago, it was 2015, we actually solved heads-up limits.
So we computed the perfect strategy. So in that case, it takes a massive amount of computation before you play, but then as you're playing, it actually just knows exactly with every situation with what probability it should fold, call, or raise.
And it just looks it up in a table. So it plays super fast because it's done all of its thinking before it sat down to play.
Well, and so given that, does that mean that it wins every time or statistically, you know, out of 10 games, how often does it win or out of 100 games?
Typically, what we've seen is that if you play against some of these super strong programs at heads-up limit... against really top players, poker players, you're going to see margins of 5% to 10%
Big blinds per 100. That number is essentially five is about the number that a professional player would like to win by in order to sort of be a professional player.
It's a healthy win rate. So we're winning against pro players by about that much, sometimes even more.
So that's heads up limit. And then what was the critical part for heads up no limit?
Yeah, so the heads up limit, the game has 10 to the 14 decision points.
So when you're thinking about all the possible decisions you could find yourself in, and you had to pre-compute them in advance, there's about 10 to the 14 of those.
That's a little over a million, million decision points.
It's a big number, but you can actually get a computer to think about all of those.
The number of decision points and the variant of Heads Up No Limit we played is 10 to the 160th.
Wow. So that's more than there are atoms in the universe.
That's pretty close to the size of the game of Go.
We're not going to reason about all of those before we sit down and play.
We can't count all of those. We can't store the strategy.
So that style of how we approach the game is totally not going to work in No Limits.
Now, the way we built No Limit programs was we still took that style.
What we would do is we would pretend two situations that we know are different.
Like, oh, I could be holding aces here or maybe I'm holding kings.
I'm going to actually push them together.
I'm going to pretend that they're in fact the same even though they're different.
And I'm going to do that a lot. I'm going to take I'm going to do that so many times that the number of decisions in the game goes from 10 to the 160 back down to like 10 to the 14.
So I can think about it ahead of time. But if you do that in a game of Heads Up No Limit, that's putting a lot of things together, right?
That's taking all the atoms in the universe and collapsing them into one atom.
Even worse than that. This is really putting situations that we know are different and we're going to have to pretend like they're the same.
And that causes our programs to make mistakes.
And that's why, you know, the strongest Heads Up No Limit programs, you know, a year ago weren't able to beat pro players.
DeepStack has to take a fundamentally new idea, and that's what's behind it.
It actually doesn't do all of its thinking before it plays.
It does most of its thinking while it plays.
And this has the advantage that it can focus its thinking on the situation at hand.
So it treats each of the 10 to the 160 decision points as being completely different and because it's going to figure out how to play that decision uniquely and doesn't have to pre-compute it ahead of time.
Describe a game for us. I mean, probably most of us have played online poker.
But from the computer's perspective, how does it play a game?
So it plays it much like a human would. It's dealt two cards, it sees its two cards, and when it's turned to act, it gets to decide to make a bet or raise or call or fold.
And so the premise of it playing back and forth looks similar.
Because online poker is so popular, it means that we also don't have to automate.
There is no dealer sitting with actual cards being dealt out and no camera system to figure anything out.
There's an automated dealer that's going to randomize the cards independently from our bot, and then the bot is only informed of its cards it's holding, just like the human interface is only told its cards it's holding.
So it basically plays through much a similar interface that you would in an online environment.
But it's now not computing the end sort of result because there's no way it could do that.
It's thinking as it goes. And what was the key to that?
So the trick is that if you're near the end of the game, this isn't so bad.
You could actually reason through all the ways this game could play out and and actually you know fairly quickly because you don't have much time like you have about three to four or five seconds tops to make your choice.
So when you're near the end of the game, like if all you have to do is your opponent just went all in and your only choice is to fold or call, You know, it's not that hard to reason through, well, if I call, well, what's the distribution of hands he holds, blah, blah, blah.
And you can sort of calculate what the right answer is.
If you're a little bit further away from the end of the game, it still may be feasible to do in three or four seconds.
But if you're near the beginning of the game, now the number of ways the game could play out is infeasible.
I just turned from one impossible problem of enumerating all the situations and computing the strategy in advance, To another impossible problem, which is do it on the fly in three seconds.
So the trick here is that we're not going to do that, of course, because we're not going to look all the way to the end of the game.
We're only going to look small number of actions ahead, And then we're going to stop.
We're going to be at some new poker situation.
And we're going to evaluate that using what I call deep stacks intuition.
I'll have to say more about this, but its job is to take a poker situation and say how valuable is it to find yourself in that situation.
And basically the program now pretends like that's the end of the game.
I was going to say that. I was going to say, so it does kind of break it down into discrete parts then and kind of solves for those parts.
DeepStack only sees a small game. It only sees a few actions ahead and then thinks the game's going to be over.
And then after it takes those few actions, it finds out, nope, the game's not over yet because now it's going to look a little deeper. and then use its intuition to answer what happens after that.
And so at every step along the way, it keeps seeing the small game. and then is able to reason, even though the game is gigantic, it's able to see a small game that it can solve in three, four seconds.
Now the trick is that intuition needs to be good, right?
It needs to actually summarize what's the value of playing out the hand from here.
And much like human intuition, we're going to have to train that.
We're going to have to, through experience, build up that intuition.
And so what we do is we have deep stack play many, many poker situations against itself.
These are now going to be much smaller situations, ones that aren't the full scale of the full size of the game.
And it can learn to play in these simpler poker situations and then when it's in some poker situation it hasn't seen before, the idea is that it can generalize. from what it learned to be able to play that.
And we're going to do that with deep learning. tell me what then in your book intuition is as it is, um, manifested in this and in deep stack, is it, it's experience, um, plus what, or how should we describe it?
So the definition of intuition I like, and obviously we don't quite know really strongly what's going on in the human brain when we're being intuitive.
But we do know that it is not like a conscious reasoning process.
We can often reason through things where we think through different outcomes and apply our logic to them.
Intuition is something separate. It's like a sort of gut instinct where we immediately react with an answer without having to think hard about it.
And there's two stages to deep stacks thinking.
There's the sort of reasoning stage where it thinks, here's my actions, here's my opponent's actions, how could they respond, here's how I could respond to those.
And then eventually it cuts off that reasoning and says, nope, I can't look any further.
I don't have enough time or computation to do it.
Let me just throw this off to my fast processing. my quick answer solution that is probably not going to be perfect, but it's going to give me an approximate answer based on all this past experience. and give me a quick answer to how valuable it is to get there.
And so I'm using intuition to contrast it with the sort of conscious reasoning process that happens for the next few actions, which then gets the rest of the game gets summarized by a quick fasting, quick reaction process.
That's the intuition. Crazy. I mean, the idea that you can teach a deep neural net to do that is just fascinating.
And was that obvious that that was the way to go?
Clearly it wasn't. I mean, it certainly wasn't obvious.
But how did you get there? We actually, we had two visiting PhD students from the Czech Republic came to work with the Computer Poker Research Group. here at U of A and spent a year with us.
And they pretty much came with, they thought that that would work.
And I've worked with them before and I've seen their work.
And so I know they're brilliant. So I was happy to bring them on.
And we had our own directions we thought was going to get us. to Heads Up No Limit.
And so I was happy to explore another one, but I was challenging them with them every step along the way to new hurdles to prove this would work.
And what I saw the problem was is what are these – like it needs to come up with a value of a poker situation.
And what are these poker situations? They're more complex than you might think.
They include things like, how has the betting gone to get here?
They include what cards have already been flipped over.
So there's the obvious things, but they need to also include what's the distribution over cards my opponent could hold?
It actually has to include my beliefs about what the opponent could hold.
Even harder, that poker situation includes what I believe my opponent knows about my cards.
So we're really when we feed a poker situation into deep stacks intuition, it has to have it actually encompasses the players beliefs about what everybody knows.
And so it's not about evaluating like a single board position in Go, which we've already seen with AlphaGo that deep neural networks can do that extremely well.
Here, it's like it's evaluating distributions, like whole different worlds you could be in.
Like what if I was holding these cards and the opponent was holding these cards?
It really has to take that into account and what the beliefs are.
And that just felt harder. That just felt like, okay, I don't know how we're going to get that.
How is the deep network? Going to be able to build a representation and find a representation for encompassing how those beliefs work.
Would you describe how DeepStack plays as relatively human, or is it something different?
And And also, has it changed how human players play the game of No Limit Texas Hold'em?
Yeah, I think it's hard to say how humans play this game.
I think it is much more of an... has an intuitive feel to the game than a game like Chess or Go has.
But with that said, I think you sort of already described my position on it, that humans can't search all the way to the end of the game any better than deep stack can.
Um, And so the limitations that DeepStack is considering in order to come up with an answer for this I think is very similar.
It's explicitly reasoning about what it knows about its opponent's cards.
It's explicitly reasoning about what the opponent knows about its cards.
And this is exactly how good poker players think about the game.
They're not really trying to put the opponent on an exact hand.
I mean, obviously, if the opponent gave their hand away, they would use that information.
But they have to compare to the whole distribution.
They always have to be thinking that way.
So that element is very similar. And then they're going to look through like a sort of how should I play out this round, essentially, like maybe I should. check raise this round which is sort of i'm just going to make i'm not going to make a bet at first but if they bet i'm going to re-raise in a sense to try to trap them Or maybe I should just check it all the way through and see what the next card is.
And so they're sort of thinking about, and this is what DeepSec does, thinking about some very sort of short-term ways of playing.
And then somehow they then have to summarize how they would play after that.
And that's where deep stacks intuition is coming in.
And I think there has to be some similar process for humans.
So I think on some intuitive level, I think this is closer to how humans play.
You ask then, does it play like a human does?
That's a whole different question. I think on some level, it might be doing those calculations better than a human can.
Like they're very similar style of doing it, but maybe it's able to – I think some of the reasoning processes I think it can do better. than a human can.
I suspect our intuition isn't as good. We actually trained it on a little over 10 million poker situations.
A poker player who's played more than 10 years has probably seen more poker situations than that.
And so it might be the case that our intuition – now we can improve that.
There's lots to be improved with let's feed it more training data.
And just see how good we can get the system.
So I think there are advantages and disadvantages of the current deep stack over what humans can do.
You mentioned the current deep stack. There's also other poker playing systems out there that play no limit Texas Hold'em.
Liberatus is the one that comes to mind.
Are you dramatically different than their approach?
And is there any chance of a... Computer versus computer, AI versus AI match.
On a fundamental level, this notion of intuition and limiting how far you have to look ahead Um, that is in fact, you know, new to deep stack, uh, and is not a piece of Labradus and it's a, in my opinion, it's a big deal.
It's what allows you to, one, get the computational burden down.
So Libratus plays with a supercomputer behind it.
And I kid you not, like there's a thousand CPUs doing computation while it plays to be able to make its decision and deep stack never has to see all that massive computation because it only ever has to look a few actions ahead and then resort to intuition.
And so not only does it typically make its actions about twice as fast as Libratus, it also can run on a laptop.
It basically needs, we did all experiments with the GTX 1080 GPU, and that's all it needs.
So a gaming laptop can run all of DeepStack. uh and and beat uh poker pros uh just just running on a laptop wow could you i guess you could because you guys are both playing the same game you could certainly play against each other Yeah, we'd be open to that.
I think there's a couple of problems with figuring out how to make the competition fair and sort of how do you set the computational limits or the time limits to take actions.
You know, in what format of the game are we going to play?
But I think also there's I think there's a bigger challenge that it wouldn't be cheap. especially if you're running a supercomputer to run a bunch of hands on one side of it.
And so I think part of it is, is there value in, in determining that.
You might think so because maybe you could benchmark the two techniques against each other.
But this is the last point is that I'm not sure it would mean what I think everyone wants it to mean.
I think everyone wants it to mean like which program is better.
Right. Um, And if you were playing a game like Chess or Go, that would be a very sensible thing to do.
If you played two programs against each other, The winner kind of by definition is the program that made fewer mistakes because every mistake means swinging the game from being you were in a winning position for yourself, but you made a mistake and now it's a winning position for them.
And now for you to win, it would have to be swung back.
And so they would have to make a mistake.
And so whoever makes the fewest mistakes would win.
In poker, because it's this imperfect information game, because it's this hidden information, the game doesn't quite work that way.
The best I can liken it to is imagine the game Rock, Paper, Scissors.
So sort of the standard children's game.
We actually all know what perfect play means in this.
We would just be totally random. If we were totally random, there's no way the opponent could beat me on average, right?
They would just break even, you know. with equal probability.
So, but what if we had some computer and somehow this game was harder than it appears?
So the computer was struggling to compute that optimal solution and One computer program tried to compute it and plays close to random but plays rock a little too much.
Another one plays close to random but plays paper a little too much.
If you play those two programs against each other, the one that plays paper a little too much is gonna win, but it doesn't really mean that it's closer to the right answer.
In fact, could play paper all the time right a dreadful strategy that you know any human who sat down to play against it would would slaughter it and you know after just like 10 playings And it would still actually beat the program that plays rock just a little too much.
So there's something slightly funny about these imperfect information games that means that if you have static programs, which both DeepStack and Libratus are, Playing against each other doesn't really measure how big their holes are.
It measures just do they happen, does one program happen to exercise the other program's holes.
So maybe a match will happen. That's still a possibility, but I don't think it's on the horizon as being planned at the moment.
What can what you guys have done with deep stack, how can that be applied to other scenarios outside of the poker world altogether?
What do you guys think about and how do you as a computer scientist think about this problem more broadly? how you can bring AI to bear in different situations.
Of course, as scientists, that's our fundamental goal, right?
As fun as it is to play around in the space of poker, and it's cool to sort of see this in a popular game.
Our main goals are to advance AI, and poker has a certain set of properties that make it both challenging for AI, but then is going to push the development of AI in new directions.
And that's what it's done over the last 20 years of work on computer poker.
You're right. You've hit on the exact point that our everyday decision-making, it's an incredibly rare moment that we have all the information we need to make a decision.
Like that's not how we go about our daily life and yet humans are able to cope with that quite readily and But our computer programs, when we look at how AlphaGo works, it assumes I know the exact state of the board and I'm going to do a search from that position.
And that's exactly what doesn't work in the game of poker.
And so on one level, yeah, we're absolutely just inventing fundamental new algorithms that are going to work. you know, advance our AI's abilities to cope with these kinds of uncertainties in situations that you don't have all the information.
Now there's particular examples of those settings that are natural applications for this.
One is security. So how do you deploy your security resources to defend some strategic infrastructure?
Like there are in fact AI systems that schedule air marshals on airplanes throughout the United States.
And the goal of doing the scheduling is that there's this nefarious adversary who wants to exploit any hint of pattern in your play.
And this comes back to, you know, I want to hide my cards, so I'm bluffing and all these other things.
That reasoning is the exact same kind of reasoning.
How do I come up with a schedule and a way to schedule?
that means that my adversary can't exploit my tendencies.
So what's next for you in DeepStack? What are you guys up to in the poker playing world next?
Yeah, so one of the exciting things that we want to do has to do with DeepStack's flexibility.
So because DeepStack doesn't solve for every situation upfront and actually plays the game as it arises, It can actually play the form of poker that humans, even in a heads-up case, is more naturally expecting a no-limit game to work.
So when we ran our studies and we actually played a slightly different variant where after every hand we would just reset everyone's stacks and This is great from a scientific point of view because it means every hand is sort of an independent sample of some quantity you want to measure.
But it's not how... how poker players sort of expect the game to go if i sat down and on the first hand i took half your chips then on the next hand you got to play with half as many chips And so that's known as sort of a freeze out tournament.
Basically, you don't get to put more money on the table.
You're going to play until one person is out of chips and the other side has all the chips.
DeepStack can play that form. Actually, in fact, unchanged.
We don't need to even train it differently.
Its intuition will be a little bit wrong in places because its intuition will be trained on a certain size of stack.
And when it's in a different situation, it'll be wrong a little bit.
So we could improve it by adding some training there.
But it actually plays the game quite well even without that.
And so one of the things we're going to do is actually run some freeze-out tournaments because they're super fun to watch.
This is kind of the form you expect to see on TV.
Where when someone pushes all in, they're about to lose all their chips kind of problem.
So we're going to run some freeze-out tournaments against some poker pros.
We're actually going to stream those matches live so people can watch.
We're going to get two poker players from the Thinking Poker podcast.
Nate and Andrew, they're going to sit down and play, and they're going to chat with each other while they play and talk through what they're thinking and what they're thinking that DeepStack is holding.
And one of the other things we're going to do is DeepStack's going to show its cards after every hand.
So normally in a game of poker, if one side folds, you never know what the other side was holding, but DeepStack's just going to always flash its cards uh and hopefully infuriate the humans uh when it bluffs it off uh the first one is going to be march 29th it's going to be at 5 p.m eastern time so just in a couple weeks Great.
Well, we'll look forward to that and look forward to watching DeepStack take everyone's money.
I want to thank you for your time. Michael Bolling, keep up the good work, and I will promise to never play you in poker.
Thank you, Michael. This was super fun. PostScript here.
If you are listening to this podcast on iTunes, Google Play Music, or whatever your favorite podcast player is, do us a favor and leave us a review.
Thanks for listening.
