Selection Bias in Variance Calculations
COAUTHORED BY ROBERT WELLS
I am a huge fan of the PrimeDope variance calculator. In fact, one of my favorite pastimes is to go on a huge downswing, plug the downswing into the calculator, and then figure out exactly how unlucky I am.
For a while, this was a very educational hobby (and also an expensive one). But eventually, I got the genius idea that something may be wrong with the way I was doing these calculations.
The main reason I became suspicious was some of the answers I got just seemed ridiculous. Extremely improbable events, according to the calculator, were happening way too often in my experience, and in the experience of other players in my network.
For example, I have seen multiple players go on downswings of 50-60 buy ins (with graphs to prove it). These were very good players with high win rates. According to the calculator, here is the probability of a 7 bb/100 winner going on a 50 buy in downswing in 50,000 hands...
Inputs
Win rate: 7 bb/100
Standard deviation: 100 bb/100
Observed win rate: -10 bb/100
Number of hands to simulate: 50,000
Results
From the PrimeDope variance calculator.
According to the calculator, the probability of running at or above the observed win rate (-10.00 bb/100) over 50,000 hands with a true win rate of 7.00 bb/100 is 99.9928%.
100% - 99.9928% = 0.0072% or roughly 1 in 14,000
In other words, there is only a 1 in 14,000 chance of going on a downswing worse than this over 50,000 hands if your true win rate is 7 bb/100. I personally have never gone on a downswing nearly this big (I also haven't played very much), but like I said, I have seen it happen to multiple players.
Now, of course the variance calculation isn't a perfect model. One of the key assumptions you make when you do a calculation like this is that a player's win rate is constant. We know that can't be true, because the player's opponents change all the time. Also, in the real world, people tilt. People play worse when they are down tons of money, which makes extreme downswings more likely to occur.
But even if these calculations are in only in the ballpark... 1 in 14,000? Come on. That can't be right—can it?
As it turns out, the calculations are correct. The problem is the way I that I was looking at them.
The odds of this particular downswing really are 1 in 14,000, but only if you play exactly one 50,000 hand sample. Obviously the players I talked to have played way more than 50,000 hands in their lifetimes. Therefore, they are much more likely to go on a huge downswing like this one simply because they have had so many more opportunities to do so.
Introducing Selection Bias
Imagine I told you that I was flipping a coin the other day, and I flipped tails ten times in a row.
"Wow, ten times?" you ask. "The probability of that is about 1 in 1,000."
"I know!" I respond, "It was so crazy."
The next question you should ask me is, "How many times did you flip the coin that day?"
"Oh... about four or five hundred," I respond.
Well, that makes it a little less spectacular doesn't it? The odds of seeing ten tails in a row are much higher than 1 in 1,000 if you flip the coin five hundred times. It's interesting, sure, but it's not 1-in-1,000 interesting.
The way I initially presented the coin flip story is an example of a selection bias. This is a mistake that poker players (including me) make all the time when they do variance calculations.
The point is that extremely improbable events are much more likely to occur when you give them many chances to happen. Poker players play hundreds of thousands of hands per year. Each hand can, more or less, be looked at as the start of a new sample.
So instead of asking, "What are the odds of going on a downswing of x buy ins over y hands?" a better question to ask is, "What are the odds of going on a downswing of x buy ins at any period over all of the hands I have played in my career?"
Primedope gives some additional statistics on downswing variance, but unfortunately they can't be used to answer this question. So I asked a friend who is a lot more capable than I am to run some simulations. All credit for the variance calculations below goes to Rob Wells. (Thanks for your help!)
Downswing Simulations
Here's how the simulations worked: 10,000 theoretical poker players were simulated, each with the same win rate and standard deviation. Each played the same number of hands. The largest downswing each player endured over his entire sample was recorded.
The simulation was repeated with four different win rates. Here are the results for a 3 million hand sample size, i.e. 10,000 players playing 3 million hands each. (We chose 3 million hands because we thought it was a reasonable estimate for how many hands a professional online poker player might play in a career.)
Average Maximum Downswing. On average, this is the worst downswing you should expect to see in a sample of this size.
90% Confidence Interval. You can be 90% confident that the maximum downswing will not exceed this value.
95% Confidence Interval. You can be 95% confident that the maximum downswing will not exceed this value.
For example, the average maximum downswing for the 5 bb/100 players was 7,000 bb, or 70 buy ins. This means that some of the players experienced a larger downswing than this, and some of the players experienced a smaller downswing, but the worst downswing they experienced on average was 70 buy ins.
The average maximum downswing for the 10 bb/100 players was only 4,300 bb, or 43 buy ins. That makes sense, because a player with a higher win rate should experience less severe downswings on average.
But again, remember that these are just the average results. The 90% and 95% confidence intervals show that the less probable downswings can be a lot worse. For example, at a 95% confidence interval (i.e. two standard deviations from the mean), a 10 bb/100 winner can be expected to go on a maximum downswing of up to 66 buy ins.
Here are the results of the simulations for three other sample sizes: 1 million, 250k, and 50k hands, respectively:
I think it's useful to present the simulations with several different sample sizes because it makes it easier to intuitively understand how frequently these downswings should occur.
For example, let's say you are a 7.5 bb/100 winner over your entire career, and you go on a 30 buy in downswing. If you look at the results for the 250k hand sample size, you can see that the average maximum downswing for a 7.5 bb/100 winner is 31 buy ins.
That means your downswing is the worst you would expect on average over a period of 250k hands. So if you play about 20k hands per month, then you should expect a downswing this big to occur about once per year on average.
Of course, you might not experience a downswing that's this bad, but you also could have a downswing that's much worse. At a 95% confidence interval, the maximum downswing for this sample size is 59 buy ins. A downswing this large is unlikely, but certainly not impossible.
In general, poker players massively underestimate variance. Most of the 5 bb/100 winners I talk to don't "expect" to go on a 40 buy in downswing, but they actually will experience a downswing about this large on average every 250k hands they play.
For high volume players, that could mean they have to go through something like this once every four months, more or less. If this terrifies you, then you're probably not properly rolled for the games you play.
That being said, I'm not going to end this article with some arbitrary bankroll management guidelines. You have to decide for yourself how much risk you're willing to take on. My goal is just to give you a better understanding of not just what is possible, but what should actually be expected.
If you're going to make poker into your career, then get ready for a bumpy ride.
