Today my favorite writer – The Sports Guy – wrote an excellent column about Bill Belichick’s “reckless” call to go for it on 4th and 2 when his team was up by 6, on their own 28 yard line, with just over 2 minutes left to play. Much of the article goes into great detail on the problems inherent with relying on statistics in such a situation (see: Insane angle #1: Statistically, it was right move). But in Insane angle #2, he can’t help but pull out his own statistics to “justify” why the Pats shouldn’t have gone for it.
Unfortunately, The Sports Guy made an all-too-common mistake while doing so – providing a reminder that however dangerous putting blind trust in statistics can be, the problem is that much worse if you don’t understand them properly.
His argument was simple. Indianapolis had already completed two long touchdown drives in the 4th quarter. By punting, New England would have forced them to do it a third time. So, to “prove” his point, he asked someone to crunch the numbers on “the number of times a team started and completed three touchdown drives in the fourth quarter to erase a double-digit deficit and win an NFL game since 2005.” The answer he found was 4 – it happens less than once per season. He then started banging his head on his desk.
It sure looks like a perfectly reasonable, statistically-based argument, but there are some major flaws. The one I’m going to focus on here (another big one is switching from %s to a raw count of a known rare situation, which is almost always an easy but meaningless thing to do) is tied to what’s called the “Gambler’s fallacy” – the belief that deviations from expected behavior in the past are likely to be evened out by opposite deviations in the future. The common example is coin flips, but I’m going to use a basketball analogy – since that’s the Sports Guy’s favorite sport.
Let’s say your playing the Cleveland Cavaliers, and for no obvious reason whatsoever they run a play to get a three-point shot for Shaquille O’Neal – who has only even attempted one such shot in the last decade or so. The defense is so confused by this that they foul him, and he steps to the line. Even though he’s only a 50% FT shooter, he hits the first two. What are the odds of him hitting the third?
The correct answer is 50%. The fact that he’s just hit his first two has absolutely no bearing on the probability he’ll hit his next one. But if you follow the underlying logic of the Sports Guy’s argument, you’d determine the answer is 12.5%. After all, that is the odds that he hits three free throws in a row. But that number goes out the window, because the first two have already been made.
That’s the Gambler’s fallacy in action, and underlies the logic of the Sports Guy’s argument. He tried to use statistics to argue down the probability of an event happening, based on that same event happening twice before. Marching 70+ yards down the field may be difficult, but just because you’ve done it twice already in the quarter doesn’t make it more difficult the third time (and things like “how many times it’s happened to three times to win a game, etc., etc.” don’t matter much at all). If anything, it would be the opposite – and I’ve got the stats to back it up.
Just kidding – I don’t really. But since I’m writing, there is another interesting thing to point out here in terms of common statistical / analytical mistakes.
The question of the day is whether the team should have gone for it. But in his analysis, the Sports Guy brings in the particular offensive set the coach chose. Now it’s perfectly valid to question this in terms of “does the coach suck?”– but from an analytical standpoint, you really should keep the two decisions separate. Whether he made the right play call for the 4th down conversion and whether he should have gone for it on 4th down are two very different things.
Why is this distinction important? Well let’s say you’re watching a game where the coach for one team is absolutely terrible. Andy Reid in any given 4th quarter will do. On third down and six, for whatever reason, he opts not to block the defensive team’s best pass rusher, who breezes in untouched for a sack. It’s now 3rd and 12. In turn, using the above logic, you can argue that he shouldn’t have gone for it on third down, because the probability of success was so low, given the play call. But it’s not the strategy that was wrong – it was the execution. Or for a non-sports analogy, hitting on the single girl at the bar might have been a good strategy… but throwing up in her drink was poor execution. Hitting on the girl with a huge fiance standing beside her was probably a bad strategy, regardless of execution. Different things, different lessons to be learned.
The reason I’m writing about all of this is because we see variants of this situation come up all the time. Arguments for why a person, team, or company should or shouldn’t do something rage, any many people throw lots of stats and other “facts” behind them. Some have bad information. Some have information from biased samples. Some interpret it incorrectly. Some confuse strategy with execution. Some make too much from too little, others too little from too much. It’s really, really easy to do – but from my experience it usually stems from trying to over-complicate things.
In cases such as the Pats game (with so many variables in play, and a low relevant sample size to draw from), I prefer simple, back of the envelope type calculations. My “best guesses” at the percentages indicate going for it was absolutely the right call. And personally, I believe the long-term benefits to the team if they had made it (demoralizing the opposition after showing them no “respect”, increased confidence, etc) outweigh the downside here (where the coach, not the players, shoulders most of the blame). I could be wrong – particularly since you can’t really quantify the latter at all.
But what I think really doesn’t matter. To repeat the important lessons: thoughtful analysis can often be more important than the underlying data for complex circumstances such as this one (in this case, most of the Sports Guy’s best points have no numbers tied to them whatsoever). And if you want to respond to a poor use of statistics with your own statistical analysis, you should probably make sure you get it right.
