Near the beginning of preseason, a Harvard Sports Analysis Collective (HSAC) study projected each NFL team’s percentage chances of making it into the 2015 NFL playoffs.
The study reached numbers that appear to carry the credibility of tested data and analysis. Because of this, along with the school name and the study’s use of assessments from Pro Football Reference and statistical behemoth FiveThirtyEight, it garnered a lot of attention.
Unfortunately, many of its numbers are heavily flawed. (I’ve compared them in here with better playoff chances in part I-covering teams 1-10; II-covering teams 11-20; and III-covering teams 21-32, and will look at both during the course of the season and run a comparison at season end. Anything can happen, but barring a statistical aberration, the Harvard Collective’s study numbers will fare worse.)
The HSAC study made several compounding assumptions. And not only did this lead to some results that may not represent the best assessment of that team’s actual playoff chances, it also led to some statistically questionable, and even unsupportable ones.
For instance, it pegged the Seahawks at a ridiculous 99% to make the playoffs. (The Seahawks were originally 95% to make the playoffs – still too high. But apparently to normalize outcomes so an average of six teams from each conference would make it into the postseason each year, their number was adjusted upward.) There’s far too much variability, uncertainty, as well as general parity in the NFL for any team to have a 99 out of 100 chance to make the playoffs, before the season even begins.
The HSAC study also pegged the Titans at 2% to make the playoffs, and originally the Ravens at 9%. Both of these are also unrealistic given basic NFL variance in the case of the Titans; and in the case of the Ravens, also given the fact they have made the playoffs 6 out of the last 7 seasons, and have more playoff wins than any team in the entire NFL since 2008; the year quarterback Joe Flacco entered the league and John Harbaugh became their head coach.
And it pegged the Raiders at a ridiculous, almost ludicrously low .003 (.3%) – that’s 3 out of 1000 times – chance of making the playoffs.
Along with a few other probabilities that push the boundaries of statistical reasonableness, and several others that probably don’t represent particularly great assessments, the study also pegged the Miami Dolphins as having the highest chances of making the playoffs out of the entire AFC.
That’s not a totally wacky pick. Miami was one of my dark horse teams to take the next step this year; as it was for several other people. But it still seems a little odd that since this study has come out, Miami is now often thrown into the mix of AFC, and even possible Super Bowl contenders.
There’s a good chance this is simply a coincidence. After all, Miami as a dark horse team (among several) was not a novel idea. They have some good players, a potentially excellent quarterback, and showed occassional signs the last two years of being able to play at a near elite team level. (Albeit several teams have. For instance, watch out for the Chiefs this year as much as if not more than the Dolphins. Another AFC dark horse that may surprise, if that defense really pulls together and QB Hoyer throws as accurately as he did the first half of last season and not the second half, is Houston. The Bills are also at least on par with Miami, and probably more likely to make the playoffs.)
But it could also be that a reasonably well publicized Harvard study floating around out there, that pegged Miami as the top team in the AFC, also didn’t hurt – no matter how goofy some of its numbers upon closer analysis.
And some of its numbers, as suggested above, are goofy. For instance, pegging the Seahawks at 99% to make the playoffs defies football reality, and at least relative NFL parity and uncertainty.
One of the only ways to really show this point is for the Seahawks to miss the postseason. (Though it wouldn’t technically prove that the “99%” probability was wrong, since, though a long shot, such an outcome could still just be a “1 in 100” fluke, it would certaintly help suggest it.)
But the problem is the Seahawks are likely to make the postseason.They’re just not “99 out of 100 times” likely to make the playoffs. And no team in modern NFL history has been. Ever.
Essentially, the HSAC study used a multiple step interpretive statistics process to come up with a methodology that appears sound, but isn’t.
The study used Pro Football Reference’s approximate value statistics for players, then assessed team strength by relying on them for “core players.” But the valuations are still subjective. And more importantly, football is a team game, not a core player game.
The model results were also “tested” by running last year’s data, and comparing it to last year’s end of season FiveThirtyEight ELO ratings. But reasonable correlation with these ratings doesn’t imply the probabilities are robust; just that they may be more accurate than merely throwing darts at a random board of probability numbers.
The ELO assessments also reflect a limiting system of assumptions as well – one that tries to arrive at the “better team” in terms of overall performance, including in large part how much a team wins games by, etc. But this also doesn’t mean correlation with the highest chances of making the playoffs – just again, something superior to throwing darts at a board.
First off, some teams know how to protect leads; others how to do so and pull out close games when behind; still others manage to stay tight and lose but can win by blowouts, etc. (And even if to some extent these things can factor in to win totals, it gets heavily skewed by score differentials, what team was coming off of what games, and most importantly what actually happened in each game.) And it doesn’t take into account the odds that particular team faces – the makeup of their division, what other divisions they have to play, etc.
So rather than test the model compared to last season’s rankings, as noted above we’ll compare its probabilities to how the season actually works out for the 32 team’s ranked, as well as how it does in comparison to the non statistical generic evaluation of each team’s playoff possibilities assesed here. My prediction is that the Harvard study, although it got a lot more publicity, is going to show worse results than the assessments made here in parts I II, and III.
In addition to the fact that grading core players rather than the full team is incomplete, and that player grades, even for all players, is still not necessarily equal to a team grade, part of the study’s flaws is that grading players relative to each other in terms of win probability is also very difficult. If one player is a 10 and another is an 8 (just for a scale of comparison), what does that mean?
Is the difference between 10 and 8 that big of a gap that surrounding “non core” players, coaching ability – beyond its small reflection in that team’s player ranking to begin with – overall team chemistry, cohesion, or heart, don’t matter as much?
Of course if we can assess the general quality of multiple key positions, statistically at least we can at least get a feel for the team. (And in many of the skill positions particularly, the study’s overall ranking, even if unintentionally, will be affected by the overall quality of a team, with receivers with great quarterbacks and solid offensive lines and great offensive coordinators getting higher ratings, for instance, than if they had played their last several years on a different team, etc.)
But that’s all the study really does. Which around the middle of the pack is enough to put forth numbers that aren’t consistently outlandish, but not at the high, and in particular low, ends.
Think what you will of the Rams, for instance, but assessing them as having only a 1 in 10 chance of making the playoffs, before the season even starts, and with an upgrade at quarterback; another year for their many young players; an improving team; a good head coach; and when 12 of 32 teams reach the playoffs, is just not realistic.
For this same reason, almost all of the low end, and particularly the very low end numbers, are not just too low, but become increasingly preposterous, no matter how bad seeming the teams. Even Tennessee, and even Oakland.
And, frankly, who knows. either could be a decent team this year. (with Oakland probably having a slightly better chance, although they’re in a tougher division and face a tougher outside the division schedule, which will hurt them in the getting to the playoffs sweepstakes.).
Also notice Oakland’s pattern last year after beating the Chiefs to bolt their record up to a whopping 1-10 in week 12. They took it light – obviously – and got pounded 52-0 by the Rams, then pulled it together and back at home surprised again, legitimately beating the still tough 49ers – and doing so as large underdogs – 24-13, before then, same pattern, getting pounded yet again, and this time by the Chiefs in a rematch in Kansas City, 31-13. Then guess what. Same pattern still: They won again, and again against a good team. That is, by late last season the Bills were a very good football team, and probably taking the Raiders lightly, and on a cross country trip fell to those same Raiders 26-24. And yet after pulling out that win, Oakland continued its pattern as well, getting pounded by Denver in a season ending game, 47-14.
Again, we’ll examine the outcomes at the end of the season, but it will be very surprising if the Harvard numbers don’t fare much worse overall than the numbers given here. In the meantime, again, two sets of playoff odds for all teams in the NFL, one by the Harvard Sports Collective study, and one by this site along with some of the key reasons for the numbers given, are set forth in parts I, II, and III.