The poker world’s computer-related attention has been on the “Brains vs. Bot” match between the No Limit Hold’em Heads Up artificial intelligence program Libratus and four hapless human challengers named Dong Kim, Jimmy Chou, Daniel McAulay and Jason Les, but over at Advanced Poker Training, they’ve unveiled a little something of their own.
APT associated themselves with last year’s WSOP Main Event Champion Qui Nguyen back before the final table started, running simulations for PokerNews that predicted him winning the bracelet more than any of the other players (26 of 100 simulations), and providing him with on-the-rail advice during the final table as well as simulator training in the weeks before.
They ran a sale on a subscription during Thanksgiving week, and I signed up (always eager to improve my game). I’ve been planning to do a review of their training program (which I still intend to do, but the short version is, I think it’s well worth the price to see if it works for you) but this week they unleashed the QuiNguyen bot in their Final Table Trainer, and I wanted to give a little sample.
The Final Table Trainer is just what it sounds like, a single, nine-person table where you can select the level of skill of the other opponents (Nguyen only shows up in the Hardest level), whick of the two dozen or so advisors you want to use (and whether you want them to always be looking over your shoulder or only when you ask), and relative stack size. I was using Daylian Cain, which is a real name of a Yale professor of management and marketing—I don’t know how many there can be—and the prof’s Twitter account uses the same image APT has in their list of named players, but he’s described in his advisor bio as a “Deep Thinking. Loose Aggressive” player who charges as much as $1,600/hr. for poker coaching. So I figure I’m doing good right off the bat. At least if I take his advice.
So here’s the situation at the very beginning of the trainer. I have 300K at 5K/10K/1K. 30bb. I start in middle position. Action folds to me. Daylian’s advice is covering a player to my left with 24bb and the Mike Caro bot with 34bb. The small blind has only 9bb and ends up picking up the pot.
I raise my to 3x (a little less than Daylian suggests, but then I think he’s more of a cash player). Jonathan Little calls from the small blind, then leads out for 18K—only 20% of the pot—on a flop of —and I fold.
Hand seven is a pair of sevens on the button. Doug Hull raises from under the gun and I call, along with Caro in the big blind. Hull opens for 30K on the flop and I fold (which is also Daylian’s suggestion).
Hand 14 I get in middle position with an unopened pot (the button is on Svetlana in Seat 9) and Daylian’s advice is to raise to 33K. I only have 22bb, though and though I’d typically play this deeper, I balk, and fold. Ditto for on the next hand.
Hand 17 is my first run-in with Nguyen, but I’m fine with that because I have aces. I start with just over 20bb in the cutoff, action folds to Nguyen, who raises to 21K. I make a bet of about 1.5 times the pot and everyone folds.
I have threes and fours on the next two hands with unopened pots. Daylian suggested a fold with the former and raise with the latter, but I fold them both. On hand 29, I get the treys again. Nguyen opens for 23K and Daylian suggests a three-bet, but I fold. Four hands later it’s deuces under the gun plus one. Daylian says to open, but I don’t, with just 21bb and blinds coming up.
I open up with jacks in hand 36, and the big blind reraises to 84K. I shove (against advice), and Svetlana folds, which puts me back where I started.
My next hand, both Daylian and I agree that can be raised to 30K, which gets through with no more action.
Daylian and I get along great until hand 60, folding pairs of fives and ragged aces, until I get The Butcher in the small blind. Calle Yang (“Tight, Aggressive Pro”) has gone all in for 11bb from late position and the only player after me (“Randy Malone”) has less than half my stack. A loss drops me down to 14bb, but I can’t lose more than that. I call against orders. Yang has and the board runs out giving us both queens and eights with a jack so we chop.
I pick up on the next hand (blinds are up to 7,500/15,000/1,500) and make what the program labels as a questionable fold to a 33K raise from Nguyen on the button. Daylian, at least, thinks I did the right thing.
Hand 66 I fold in early position instead of opening to 51K (~20% of my stack). We’re still nine-handed, I have 16bb and there are four stacks with 6–10bb still to act behind me. No thank you.
I make a 3x raise in middle position with . Everyone’s still in, two short stacks have managed to chip up, and if you look to my right, you can see that the Qui Nguyen bot has the lead by nearly 150K. I take the blinds and antes. I get fours the next hand and again elect to fold rather than raise (Daylian’s suggestion.
Hand 77, Malone on my left shoves 100K and Yang re-shoves from the button for 122K. Both Yang and I fold our blinds (I have . I don’t get to see theiir cards in the replay, but Yang wins. Six hands later, Old Man Caro is wiped out by Svetlana.
I fold my way down to 10bb by Hand 96 when I pick up a pair of tens. I’m the short stack on the seven-handed table and I open-shove from the cutoff despite advice to bet a third of my stack. Nobody calls and I’m still below my starting stack.
I have on the next hand when the Nguyen-bot raises to 38K (627K behind). I shove for 189K (which the program labels “Interesting Bet — Preflop,” and pick up the pot.
I get a “Questionable Fold — Preflop” with a pair of fives on the button facing a raise to 32K from Yang on Hand 152. My Daylian says “Call” but my finger says “Fold.”
I shove as second-to-act a few hands later where Daylian says the only thing to do is fold. But it works.
Hand 107. I’m on the button with 255K and . Little’s bot raises to 32K from the cutoff. He has 470K, second only to Nguyen (with 682K), and I push back. The advice is to raise to 93K, but I push my whole stack in (only 15bb or so) and get a fold.
Four hands later and it’s a suited ace. Here I just call the raise from Yang to 36K (advice is to fold) and get a flop if . Yang bets 44K and I fold.
On the big blind three hands on and Yang raises the button to 36K. I call with and we see the flop of . I check and fold to Yang’s c-bet.
Action folds to me in the cutoff with and I shove 13bb. The button and small blind have more than I do but I have the big blind covered by three or four blinds. Daylian only recommends a raise.
I shove with under-the-gun plus one on hand 119 and get folds all around. Nguyen is still in the lead (661K) but Yang’s taken 2nd (581K), with Little in third (442K).
Hand 123 is another “Interesting Bet — Preflop” (I do like how they don’t just say: “Bad.” Yang opens to 46K with the blinds at 10K/20K/2K, and I shove . I only have half Yang’s chips but the 3-bet gets a fold.
But I fold on the next hand even though the pot’s unopened and I’m in a later position than Yang. Why? Because I don’t plan on folding if I raise.
It’s hand 127 and I open to 60K under-the-gun with and 250K behind. Everyone folds.
Hand 131 I open shove with in the cutoff and pick up the pot.
Hand 133 and I open to 60K with . Daylian says 70K but I try to keep it simple. Folds around and I’m at 400K.
Hand 142 and I get in the big blind. Everyone folds to Nguyen, and the bot raises to 48K. I call (agreeing with Daylian). The flop comes down . Nguyen bets 46K and I fold with 304K left. Fourth of seven with all the smaller stacks in a line on my left (and the bigger stacks in a line on my right).
I ignore Daylian’s advice to raise from the cutoff and fold hand 145. I shove under-the-gun on Qui Nguyen’s big blind!. Ace-high like a boss!
I do not fold on hand 159 when Jonathan LIttle opens under-the-gun to 42K. No! I shove and everyone bows to my poker might!
Yang (still in second place to Nguyen with Little well behind) opens her button to 44K on my big blind with . I have less than 14bb and shove to pick up the pot.
182 hands in and I’m down to 11bb at 12.5K/25K/2.5K. I fold a pair of fours against advice to raise. Five hands later, it’s and I open with a shove to 11bb for a win.
I fold from the small blind to a shove by one of the small stacks, then a pair of sixes on the button to another all in (and a call). I ignore advice to open for 3.5x (from a 9BB stack)with and shove instead. Yang calls with from the big blind, catching a on the turn but losing when I get on the river. The double up puts me in a close third (with Nguyen still in the lead).
I fold under-the-gun on the next hand, then Doug Hull is all in with the shortest stack (less than 5bb) on my big blind. Yang calls.
My advisor says to fold here. I feel that Yang’s is sheerly a position call, and Yang-bot has 14bb behind if it folds. I’ve got one of the strongest hands in Hold’em. I shove instead. Yang folds, the board runs out and I become chip leader with six players remaining.
in an unraised pot two hands later is advised as a raise, but I fold. Meanwhile, Nguyen has doubled up Little, who’s not in second place about 160K behind me.
Little moves up as I fold a series of hands, until hand 205, when he opens under-the-gun for a little over 2x and I push it from 58K to 140K with on the button. I have top pair/top kicker on the flop—as well as the nut flush draw—and over-shove for my remaining 531K. Little-bot folds.
on the next hand, I fold against advice. on hand 207 and I open for 75K. Yang calls. The flop isn’t as quite as good as it was two hands ago, but [6g] is reasonable. Yang checks from the big blind and I bet another 75K (less than the 120K recommended). Yang folds, and is now down under 12bb.
on the next hand and I open for 75K. Everyone folds. Hand 208 and I am the first player to one million chips with Little the closest at half my stack and just six players remaining (including me).
I raise in the next round as second-to-act, then under-the-gun. I fold a big blind to a raise and 3-bet, then in the small blind to a 60K open. Fold on the button. with a Little open and Nguyen 3-bet.
My next big blind, both the short(est) stacks shove and I have with no other players left to act. The program labels this as a “Questionable Call”, but with 800K and the chip lead even if I lose, I have to disagree. Even Daylian said to fold. And maybe they were right. as Lebedeva under the gun makes a set of deuces on the flop. Villegas has queens, but an ace on the turn eliminates him, and I pick up a side pot of 52K, though Lebedeva snaps up into the tight competition between second, third, and fourth place (less than 1bb separating the three of them).
On the button, I have and 3-bet Yang to 250K (more than the 188K suggested), The flop is fantastic: , and the nut flush.
Yang bets 220K, leaving only 24K behind and I just call. on the turn, Yang puts in the last chips, I call with the nuts, and the river brings the making a full house for Yang’s . I really don’t see any way of avoiding that result; I’ve got the nuts on the flop and Yang’s committed with the set. The loss knocks me back to 366K (14bb) and fourth place on the table. Yang takes over the lead.
My next hand is and Kayllian suggests a raise to 83K. I fold. ON hand 230, I get an “Interesting Bet — Preflop” note. I’m in the big blind with . Under-the-gun opens to 55K. Nguyen calls from the small blind. I squeeze for 308K, less than either of the other two players, but for a significant portion of their stacks. I get two folds and move up close to third.
The Little-bot shoves on hand 231 and I go against advice to shove a pair of sizes from the small blind. Little has and misses the board. I almost get a straight flush. Down to four, but more importantly, I’m in second place.
on the next hand and Nguyen opens to 63K. I go a bit beyond the recommendation and put in almost half my stack for a 325K 3-bet. Everyone folds and I have over 800K again. I raise to 75K with on the next hand and take the blinds and antes.
Nguyen raises my big blind () and I fold. Yang opens to 63K on my small and I fold : there’s going to be a lot of this coming up.
Nguyen-bot raises under-the-gun and I 3-bet him again to 265K, with . This time he calls, the flop is , he checks to me and I bet 300K (with just 247K behind). He only has 259K and calls with , then gets lucky as the turn is and river is . We split the pot.
I fold (advice is to raise to 85K), then squeeze over a 63K raise from Lebdeva and call from Nguyen. I have both of them covered, and they both fold. The win puts me back in the lead by all of 20K.
I raise the small blind to 90K as the blind level goes up to 15K/30K/3K, Lebedeva folds the big blind and I’m back to a million.
Waylian recommends a fold of my under the gun, but I raise to 90K. Lebedeva—the current short stack—shoves 232K and everyone folds.
Yang opens to 66K next, and I 3-bet with . Yang folds. I stay pretty quiet for a while, drifting down from 1 million to 850K as Yang chips up over a million. I open to 90K when Yang and Nguyen fold to me small blind, and take Lebedeva’s big.
Nguyen makes another under-the-gun open on hand 256. I 3-bet with (advice is to call) and take it down pre-flop.
On our next blinds, Nguyen opens for 75K into my and I raise to 300K, more than half his stack. That wins.
on the button. I open to 90K (with Lebedeva and Yang in the small and big blind, respectively). Lebedeva shoves (196K), Yang folds, and I call). It’s a race against Lebedeva’s but the cards go all around with . It’s me (1.29M), Yang (931K), and Nguyen (485K) after hand 260.
At this point, I’m mostly going to let these guys beat up on each other. I don’t care which one I get heads-up with so long as I get heads-up. But I do call a button raise to 63K from Yang on my big blind, with (that’s also the advice). I check call 60K (against advice) on the flop of , then check fold to a bet on the turn.
Yang’s up to 1.2M ten hands later, I’m at 1M, and Nguyen’s holding steadish at 444K. I raise to 90K from the small blind with . Yang calls and I bet another 90K when the flop is . Yang folds.
90K as a raise from the button with on hand 272 wins. On the next hand, Yang raises the button and Nguyen strikes with an all in from the small blind to win.
After Nguyen folds his button a couple hands later, Waylian advises a raise to 102K for my small blind hand of . I’m just not doing that here against the other large chip stack. I fold. I do raise in the same position two orbits later. Yang calls. The flop is . Waylian wants a c-bet of 117K, but I check, followed by Yang. on the turn and I check again (this time with approval). Yang checks. We both check the on the river, Yang shows and I take it with an ace kicker for the pair on the board.
I get and raise a small blind for a win. Meanwhile, the stacks have evened out somewhat. Nguyen is in third, but he’s pushed his way up to 765K. Yang has 837K, and I’m in the upper 900Ks.
On hand 285, Nguyen raises to 69K and I call from the big blind with . I call 60K on the flop and 78K more on the turn, but balk at the 150K river () bet of 150K.
Nguyen calls from the small blind as the chip leader on hand 288. I check and the flop is . on the turn, on the river and both checked. I win the hand with the deuces. Nguyen had [td5h].
Nguyen raises to 75K from the small blind on hand 291. I re-raise to 180K with in the big blind. He has just over 1M remaining (I have 727K behind)> The flop is and he continues with a bet of 150K. I fold.
The next hand I call his button raise of 66K from the small blind with . Yang folds. The flop is and I bet 90K. Nguyen calls.] on the turn and I bet 150K. Nguyen folds. The win puts me back in second place.
I raise a small blind to 150K with black sizes and Yang folds.
Blinds are up to 20K/40K/4K when Yang raises from the button to 100K. Nguyen folds and I reraise to 320K with in the big blind. Yang has 516K behind and folds. On a button, I open to 120K with and get folds. There hasn’t been a hand go to a flop for anyone for a while. I get a walk the next hand. Yang’s far behind with 445K. At 1.06M, I’m close to Nguyen’s 1.2M.
I fold a button despite advice. Similarly, Waylian suggests a raise to 136K in the small blind with and I ignore him. I do raise to 120K on the button. Nguyen calls, the flop is and I just let him have it when he bets 110K.
Yang shoves for 425K on hand 309. Nguyen folds, and I decide to take a shot at the knockout here with a call (Waylian says: Fold). It’s for slightly more than half my stack (less if you consider that I’m already in for the big blind. Yang has , I catch a ten on the flop, Yang has a gutshot wheel draw on the turn, and then catches the ace on the river. Back to the basement.
A couple of hands later, on the button and I shove for less than 10bb. Nobody calls.
I make the program mad by folding deuces in the small blind to a Nguyen button raise for a quarter of my stack, then shove on my own button. Yang calls from the small blind with but I get the king on the river and get back into second place, leaving Yang with only 7bb.
Yang shoves the next hand with on the button. I have in the big blind, the board runs out and this time there’s no chop. I win the hand and go heads-up with the Qui Nguyen bot.
Nguyen-bot has almost 60% of the chips, but I have the advantage of being human. The first contest of the heads-up is hand 323 (from the start of the final table). Nguyen raises from the button to 84K. I call with . I have an open-ended straight draw on the flop of , I check, Nguyen bets and I shove. Nguyen folds, I fold a couple of ragged jacks, then raise from the small blind only to have Nguyen call. The flop is garbage for me , but when Nguyen checks I shove and he folds.
I raise and Nguyen calls (by this point, I’ve pulled ahead of him by 150K). I have top pair on a board of and shove. He folds.
Two hands later I call his opening raise to 92K with . The flop is and I check fold to his bet.
I limp from the button with . Nguyen checks. The flop is and we both check. On the turn, Nguyen bets 40K. I call. on the river makes my two pair, Nguyen checks and shows . I take the pot.
Nguyen raises the button to 88K, my advisor says call but I 3-bet to 256K and Nguyen-bot folds. I have 1.57M and our positions have reversed since the beginning of the heads-up match.
I open to 160K with (about half again what is recommended) and Nguyen calls. The flop is , he bets 136K and even with an over card and gut-shot straight draw, I fold to wait for a better spot.
Nguyen opens to 88K and I shove with . Not recommended. After my pre-flop fold on the next hand, he folds to my . I’ve got about a 300K (7bb) advantage in a 67bb world.
Hand 338 I’m on the button and open black tens to 100K. Nguyen calls, the flop is and I shove. Nguyen folds.
I shove on him the next hand with when he opens for 100K, and he folds, then I open-shove with from the button. I’m up 1.7M to 1M.
Nguyen folds his button on the next hand, then I raise the next hand and get another fold.
He raises a hand and I fold, then I bet 120K on the button with . That’s also the advice Waylian gives, but the analyzer rates it as a “Questionable Bet — Preflop.” The Nguyen-bot 3-bets to 364K. Against advice, I shove, and Nguyen calls with . I double him up and he’s ahead 5:2.
I shove (advice is to call) over Nguyen’s 88K raise on the next hand. He folds. I shove and he folds. He raises my big blind while I’m holding and I fold. is an open-shove. He calls with , my nine pairs on the flop, and I retake the lead on hand 348.
I fold to a raise, then shove . Nguyen-bot calls with and it’s all over on the river.
Of course, this is not the same as actually playing against Qui Nguyen, and it doesn’t have the stress and environmental factors of a live game with $8.5M up top. I got lucky on my last two hands, winning my all-in with a 43% hand and knocking out Nguyen with only a 36% chance.
Still, I went into the heads-up with the Nguyen-bot with a plan to force the simulation of a high-variance player into high-variance situations.
I hope to do an in-depth review of APT at some point soon, but here’s a little bit about what I could learn from this particular matchup.
I’m particularly fond of this chart, which is produced for each APT session. I’m not necessarily a favorite going into a hand, but in more than half of the hands I played to the turn, I improved my equity (not the case for every one of my sessions on APT). On this particular tournament, I got an overall rating of 115 on APT’s “Poker IQ,” with 160 as the max and 100 as the mean average. And I did better (adjusted for luck) than 83% of the other players on the site (though I will point out that falls 2% short of even the 15% payout field of a WSOP event). The chart produced for each session includes ratings for your performance on each street, as well as standard session stats like VPIP and CBET%, situations where you or an opponent raised (with playback links), hands where you won or lost the most, hands you’ve tagged, and hands the analyzer rates aas questionable or interesting. You can view a sample report here.