Taking On the Harvard Sports Collective’s Zany NFL Playoff Projections

A few weeks back, a popular Harvard Sports Analysis Collective (HSAC) study projected each NFL team’s percentage chances of making it into the 2015 NFL playoffs.

The HSAC study relies upon subjective data (PFF “core” player evaluation, ELO team rankings), and makes several compounding assumptions.

Regardless of the reasons, the study reached several flawed conclusions that nevertheless have the credibility of “rigorously tested” data and analysis behind it, and garnered a lot of attention.

So just below we’ll compare the study’s assessment of each NFL team’s playoff chances with our own. (And as promised here.)

This piece will assess the HSAC study’s top ten teams. The next two will assess teams 11-20 and 21-32. [Update: Coverage of teams 11-20 is now available here, and of teams 21-32, where the wackiest Harvard study numbers reside, is available here.]

We’ll also compare both sets of numbers with exactly where each team winds up at the end of the regular season. And, to be repeated (regardless of outcome) at season end: Despite general variance and unpredictability, it will be very surprising if the Harvard numbers don’t fare much worse overall than the numbers given here.

The opening percentage number provided in bold represents each team’s chance of making the playoffs according to the HSAC study.  The ending percentage number, also in bold, is this site’s assessment of that team’s chances.

1. Seattle Seahawks, 95%.  This number is starting to close in on being statistically ridiculous. [Update: weeks after the study came out, a couple of the numbers were altered. This included the Seahawks projected chances, which, now at 99%, has reached statistical ridiculous. More on this number, an analysis of the study itself, and a few of its other more egregious examples, can now be found here. ]

While the loss of seeming top notch Seattle defensive coordinator (DC) Dan Quinn (HC, Falcons), may not hurt any more than the 2013 loss of seeming top notch DC Gus Bradley (HC, Jaguars), NFL football is not that predictable:

Earlier last year, as defending Super Bowl champions no less, the Seahawks were far back and a long shot to even win the division. They are likely to make the playoffs again this year. But giving them a 19 in 20 chance is unrealistic. Even with a 10-6 record they could miss the playoffs – particularly in the NFC West. And given that division‘s likely toughness, and possibility of some close losses or key injuries, more than 6 losses is also realistic.

My number is a guestimate, and might be slightly low; but in terms of football reality, variance, and unpredictability, 95% is almost a joke: 75% 

Note: While a drop from 95 to 75 might not seem like much, it is a huge drop in terms of probabilities, which is what the Harvard study was all about: 95% means that 19 out of 20 times on average the result will occur. So randomly we would have to replay “planet earth, NFL season 2015,” 20 times just to have the Seahawks on average miss the playoffs one time.  In contrast, 75% means a 3 in 4 probability, which means that on average 3 times out of 4 the event will occur.

Note also that looking at what happens with Seattle won’t tell much in terms of comparing the Harvard Study with the assessments made here. But examining exactly how the Seahawks and every other NFL team wind up faring – both in exact wins and proximity to the playoffs in relation to the original assessments – will tell an awful lot.

Update: The study, presumably (so it now reads) to “normalize” it’s numbers (it so reads) such that an average of six teams from each conference would make the playoffs each year, it changed a few of them, but not most. And as noted above, the Seahawks were one of those changed, and this almost silly 95% figure has turned into a fairly statistically ridiculous 99%. Again, a more detailed assessment of the study itself can now be found here.

2. Green Bay Packers, 93%. Ditto, and for much of the same reasons as No.1 above: That is, this number is extreme, and not reflective of realistic NFL variability and some degree of unpredictability.

Divisionally, the Bears, with a new HC (head coach) in the usually successful Jim Fox, along with other changes and an always potentially dynamic but also sudden error streak prone Jay Cutler, are a bit of a wild card.

On the other hand, in the playoffs last year the Lions almost the Cowboys – and but for a penalty flag that should have been called may have easily beaten them; who in turn but for an almost catch that wasn’t likely would have beaten the Packers (who then but for a meltdown at the end of the NFC Championship game in turn should have beaten the Seahawks for the right to to play in the Super Bowl).

The Vikings could also always surprise this year – and probably will to some extent.

With the Lions likely in it, and the Bears or Vikings possible contenders, the Packer’s seeming lock on the division is uncertain; it’s also unlikely more than one wild card spot will come out of the NFC North, and the Packers could be battling for that spot.

Or the whole division could be behind the two other NFC WC teams and will only send their division winner to the playoffs. And that’s without the division lagging nearly as much as in 2013, when the Packers won a tight race at 8-7-1, in a year where Aaron Rodgers missed just under half of the regular season.

Given this, and simple general NFL variance and injuries, 93%, is far too high. 80%, or 4 out of 5 is still high, yet remarkably more realistic than an almost a 14 out of 15 chance (93%), which is almost silly.

93% might not be quite as silly as the Seahawks 95% however:  Remember in the NFC championship game Green Bay went toe to toe with Seattle (In Seatle, too); and helped by a couple Russell Wilson picks as well as fortuitous bounces that happened to land in Green Bay defender’s hands, seemed to outplay Seattle for much of the game. While this season could emerge differently, the NFC South also still looks like a tougher division.

But, interestingly, the NFC North and West play each other this year. And, on the flip side (edge Seattle), the North also plays the potentially very tough AFC West, while the West plays what is as of right now still one of the two weakest divisions in football – the AFC South.

These two tough divisions faced by the NFC North also drop the probabilities of making the playoffs lower. This was the original number in the original draft however, so we’ll keep it: 80%

Note: Much of this assessment, as with most, was written shortly after the Harvard Study came out. And I’ve tried not to change them much based upon how starters have looked in pre season games, etc. (and most of that is subjective, and of minimal value at this point). The Packer’s chances though are probably also a little lower now with the loss of No. 1 WR Jordy Nelson for the season, but we’ll stay at 80%: It’s a number I originally noted was already borderline high anyway, but not unrealistic given Aaron Rodgers and the team’s perennial performance under head coach Mike McCarthy, and their position right now as the favorite based on last season’s late dominating performances. Though, frankly, taking into account the NFC North’s very tough scheduling and perhaps (now) their loss of their most reliable receiver, 80% is too high as well.

3. Miami Dolphins, 77%.  While the Dolphins blew a hot weather home game against those same Packers earlier in the year that they should have won, the Dolphins had a stretch last season where it looked like they had turned the corner and could hang with anybody.

Then they faded, as has happened before.

In 2012 QB Ryan Tannehill was also overshadowed by the remarkable QB draft class of 2012 and Andrew Luck, Russell Wilson, and at least at that time, Robert Griffin. But Tannenhill has great potential, and once again the Dolphins could take it to the next level.

Either way the NFC East isn’t going to be an easy task to take again for the defending Super Bowl champion Patriots, as the Bills will likely make the playoffs for the first time this entire millenium (quarterback problems and Rex Ryan’s seemingly somewhat random pre season handling of it notwithstanding); the Jets should improve; and the Dolphins aren’t a bad dark horse pick to surprise.

But giving this team the highest chance in the AFC to even make the playoffs, based upon a methodology that’s a nice idea as one part of an equation or approach rather than the equation as utilized in the study, is, again, ridiculous. I liked the Dolphins as a dark horse, but even my guestimate may actually be too high: 45%

4. Kansas City Chiefs, 61%. Many balked at the Chiefs being so high, and in particular being higher than the Broncos. But this is the first of the Harvard SAC probability numbers that’s not borderline ridiculous: Remember, the study is not predicting that the above teams will make the playoffs, but their percentage chances of doing so, which is where the numbers get off kilter.

Check out HC Andy Reid’s long term record: Management may have had a lot to do with it, but Reid brought his Eaglest to the playoffs most of the years he was there; and all the way to the NFC title game four times. It’s quite a record. He came into Kansas City and immediately brought them to the playoffs; then his second year (2014) they faltered, but were still a tough matchup.

The Chiefs are also getting some players back; The Broncos’ Peyton Manning was slowed late last year either by leg injury or father time; the Broncos have a new unknown in head coach Gary Kubiak (who certainly wasn’t great as long time HC of the Texans); and the Broncos weren’t dominant late last year.

It’s a tossup as of right now when these two teams play, and the Chiefs should (but may not) edge out the Chargers for second best in the division, possibly even best: 52%

5. New England Patriots, 60%.  Now we come to the first difficult one. The Patriots record in the “B & B” years is exceptional. But they have missed the playoffs before, if rarely. And during the first half of last year’s Super Bowl, Tom Brady was uncharacteristically shaky. (Though he dug deep and was focused as a laser beam in the second.)

Brady looks young, in shape, and has been still playing at a high level. But he also just turned 38. The Patriots always seem to do well after jettisoning players, but this year they’ve lost some key members of the secondary, and a few others, and it could be a change in combination with Brady’s age and some signs of a return to QB’ing mortality. (Though some of that success was also likely Belichick, and his return to mortality is probably not anywhere near age dependent at this point.)

As of right now, the Patriots will also be without Brady for the first quarter of the regular season. (Though based on an unspecified leap from concluding Brady had general awareness to specific involvement in the deflategate scandal, or that Goodell punished Brady because of an “optimistic” CBA reading of the CBA and thus granted himself the right to the entirety of a player’s private cell phone records for an on field equipment transgression issue, Judge Berman could vacate Goodell’s ordered suspension – following the same pattern as last year. Add on: 2014 No 62 pick overall Jimmy Garoppolo has shown some serious pro NFL quarterback potential, though we’re not going to change the number below.)

This year the AFC East could be tough and more upredictable than in years past, as both the Dolphins and Bills could battle the Patriots this year.  And, if he continues Rex Ryan’s “rise up and play like it’s a different game when facing the Patriots” tradition, Todd Bowles’ Jets somehow could also – at least when the two teams play.

But it’s the “Patriots.”  And that mean’s B & B’s record: That record, spanning almost the entirety of the Patriots’ Brady Belichick years as well as this new millenium, is far beyond random, and can’t be ignored. (Defending Super Bowl champs, while even playing with a little bit of a target on their back since every team wants to upset the champs, also normally do make the playoffs the following year.)

And while the Bills were solid last year and a darn good team by season end, if 2013 No. 16 overall “reach” Bills pick EJ Manuel doesn’t progress, and former Ravens 2011 6th round pick Tyrod Tayler doesn’t surprise, then “plays well when the situation is easy” perennial if solid backup Matt Cassel is probably a drop off from the shrewd game (and salary) manager Kyle Orton, who retired again.

Also, the idea that the Bills will continue or even improve upon their end of last season strength is still theory at this point; as is the Dolphins step up to that elite “you don’t want to play that team” circle – probably even more so.

With the Jets and the sometimes streaky Ryan Fitpatrick likely to be another bit of an unknown (and the up and down Geno Smith now healing a broken jaw courtesy of a silly “one guy break’s jaw of the team’s QB in the locker room” scene more fitting for the HBO football series Ballers, whose cast even would have been more appalled than Rex Ryan – who immediately signed the culprit – seemed to be) – the Patriots have to still be the slight favorite to take this division; over the Bills. With the Dolphins possibly not far behind. And who knows on the Jets.

It’ll show even more about the team, and Brady and Belichick, if as defending (if barely) SB champs, they can somehow keep it together and contend again. No controversy here, though it’s in part on the fumes of B & B’s history, we’ll almost equal the number: 64%

6. Denver Broncos, 57%  The Broncos were assessed above.

The fact that LT Ryan Clady will miss the season also doesn’t help, but Clady missed most of 2013 as well. Manning is like an on field coach, whose reads, adjustments and micro quick decision making at the line and after the snap are sometimes almost machine like perfect.

But there are too many unknowns here to pen the Broncos as a strong favorite. And their recent domination might be over. Yet on the other hand, since his rookie year in ’98 it’s hard to find a season that as the starting QB Peyton Manning has missed the playoffs. That makes this the second toughest call, after the Patriots – including the fact that it’s further complicated by Manning’s advancing football age; which will be 39 and a half, a week and a half into the regular season.

This is probably low given Manning’s record (and what a disappointment it would be for him); but without him there’s little that on balance suggests this is a playoff team. 55%

7. Detroit Lions, 57%. This one is also reasonable. It’s odd to think the Lions (who got plastered by the Patriots last November) have about the same chance of making the playoffs as the Patriots.

And this is also a tough call, as the Vikings could surprise; the Lions defense could be better, yet did lose key pieces; and QB Matt Stafford, who actually does play a lot more clutch than many QBs yet somehow also manages to both play clutch and lose a lot of close games (and almost always to good teams) – hard to do – remains an enigma. 60%

If there’s error here I’d have to say it’s to the upside. Green Bay was weaker early in the season, and the Lions outplayed them, but couldn’t hang with them (performance or score wise) when it mattered at the end of the season. Yet they could close that gap this year. And even though the HSAC Packers number was an absurd 93%, I still had it at a possibly too high 80%.

8. Indianapolis Colts, 57%. It’s not a ridiculous number, but once again, un huh.  Andrew Luck; Colts improving; and it was a cakewalk of a division last year for Indy, who is 12-0 against the AFC South the last two years.

Even though the division will likely be tighter this year, odds are that aside from its “top” team, this division is still likely to be the weakest in the AFC. And, once again, Andrew Luck, whose got heart and clutch skills no statistical core player study is going to capture. 70%

9. Atlanta Falcons, 55%.  This is too high. The Falcons have a possible good head coach coming over in former Seahawks defensive coordinator Dan Quinn; underrated Matt Ryan does remain “Matty Ice”; Mike Smith, who had done a very good job as Falcons HC, might have been burned out a little his last year; and the NFC South was very weak last season and likely won’t jump to being a monster this year.

(Plus, though we won’t let it change the number given below, the Panthers, who won this lagging division last season, just lost two starters for the year – including number one wide receiver Kelvin Benjamin now going into his second year, and the key part of an otherwise very non deep receiving corps.)

But the division is still at best a tossup right now between the Saints, Panthers and Falcons, and the Bucs could even be a bit of a sleeper this year.  (Unfair add on: Watching pre season week 3 very carefully – wherein number one overall draft pick Jameis Winston regressed – number two draft pick Marcus Mariota has the clear edge over Jameis Winston; and the Bucs, and Winston, have some serious work to do in order to make that happen.) Plus, unless things change drastically in the NFC this season, a wild card is very unlikely to emerge from the South.

If you ignore the Bucs altogether, as well as the chance of any wild card team emerging from the division (which may not be identical odds, but they at least partially cancel each other out), that leaves three teams with a roughly equal shot at making the playoffs (at least before the Panthers injuries), making anything too substantially above 33% silly.

And, frankly, while the NFC South could improve and produce wild card winners, the Bucs could easily go from worst to first in a division that since it’s inception in 2002 has only seen a repeat division winner one time (last year, the Panthers) and all four of its teams win the division an unprecedented 3 times or more. (All four have all also reached an NFC Championship game as NFC South reps; and three, a Super Bowl.) (Update: After that week 3 preseason observation, that does look less likely however.)

On the plus side, the NFC South does play the NFC East this year. The East, perhaps somewhat more unpredictable than the others at this point, is likely not an easy division but is one that, depending on how things turn out, could still be weaker than the North. And it is one that at least at this point is weaker than the still rugged NFC South. And more importantly, the NFC North also plays the AFC South – also at this point, still solidly the worst division in the AFC. That potentially ups the divisional wild card chances a bit, but probably not enough: 42%

10. New York Jets, 51%. We’re in the middle of the HSAC probability predictions, and the middle tends to mute the extremes a little, so few of these are as bad as some on the higher and lower ends. But this one is also very high.

The Jets have been all over the place. Sure, now that Geno Smith will be gone for about half a season (this happened after the HSAC study), this gives more knowledge. But Smith was up and down, and Ryan Fitzpatrick can play pretty well at times. And if Fitzpatrick stays hot the Jets should keep rolling with him: While if he falls south for two games in a row or badly so for one, given his prior history the Jets should immediately plug in Geno after week 8, who will also have less pressure this way. So the loss of Smith may not be a big deal.

Some years back new Jets HC Todd Bowles seemed to do a good job as interim HC for the Dolphins in his only, if extremely brief, head coaching experience.. But he didn’t see much improvement early when he took over as the Eagles defensive coordinator from a much maligned Juan Castillo:

Castillo perhaps should have been fired after the 2011 season. But the Eagles defense improved under him early in 2012, yet he was then fired and replaced by Bowles after week 6 of the 2012 campaign anyway. Bowles, in turn, then went to the Cardinals for 2013 and 2014, where his defenses did a great job keeping points off the board.

General guestimations are that Bowles will be a good head coach, and those guestimations are shared here.

But the Jets are still a fairly big unknown; Rex Ryan may have gotten his team to overperform a few times last season (although it’s hard to assess; this season and next will tell more about both coaches); the Dolphins and Bills should both be better or just solid; and at this early point several possible AFC wild card contenders ahead of the Jets still stick out. So putting their chances of being one of the 12 out of 32 teams who dances onward past week 17 at 50-50 is very iffy.

Emphasizing that potentially very strong Jets defense (who appeared to have added another stellar piece in number 6 overall pick Leonard Williams this past spring), positive speculation on Bowles, and not last year’s miserable performance or the Jets history of missing the playoffs for several years now: 38%. (Though if Bowles gets that entire defense – now with Darrelle Revis back at CB – playing monster, it will be higher.)
_______

We could give a lot of reasons why the HSAC study is off. [Update, again a more detailed assessment of the study is now found here.] But one key ingredient that even a better study can’t integrate – hard as it is to measure, subjective though it may seem to be, and not to sound like Gene Hackman in the great football flick “The Replacements” – is heart.

The Harvard study, by focusing on the “core” players of a team to assess value, misses that critical full team element, including the contribution of less marquee but still starting players, whose strengths or weaknesses can play a critical role in a team’s results; the effect some players can have on others; and it misses heart.
_______

[Update: Assessments of teams 11-20, and 21-32, can now be found here, and here.]

Advertisements