The Hidden Challenge in NFL Draft Evaluation

(Last updated, 6-26-15)

Drafting is both art and science. The goal is to select players most likely to help your team, and help your team as much as possible.

The problem with this seemingly simple “science” is two fold.

The first problem is the difference between real and perceived college performance. In other words, how a player seems to “perform” in college is given too much weight in terms of evaluating how well that player actually “performed.” We’ll come back to this idea.

The second is that there is insufficient weight given to college performance (and again, real performance in particular – not perceived college performance), relative to other factors which are sometimes given too much weight.

That is, factors which can be separately and precisely measured off the field, but that may already be factoring into actual on field performance, can easily be given too much separate weight or importance in lieu of or in addition to actual performance. And many of the most common of these, consist of NFL combine “measurables”

In considering these “measurables,” it’s human nature to under estimate or insufficiently consider how much some are already integrated into a player’s on field performance. This sometimes leads to over evaluation of a player’s real football abilities based upon both on field performance as well as measurables when the latter – measurables – may already be playing a large role in that on field performance in the first place.

In other words, the player performed well on the field, or at least as well as they appeared to have, in large part because of those measurables, not separately from them.

Thus giving too much weight (or sometimes almost any weight at all) to measurables can sometimes result in a form of “double counting” – giving higher on field grades due to these same measurables, which are then “added’ as separate “plus” factors to the players grade. Whereas on the other hand, better inherent football player skills, in the absence of such strong measurables, are only being single counted.

An example should make this crystal clear. Then we’ll look briefly at the first factor – actual evaluation versus perceived evaluation. (Which, as is the case with the over “measurables” problem is one that also hinders proper NFL player evaluation, and plays an enormous role in who to sign, re sign, or not sign. And, for how much to sign them for.)

A Cornerback Comparison

Consider two college cornerbacks.  Both tackle fairly well, with,at least “reasonable” execution. (Poor tackling technique and execution is the single most important and improvable aspect to improved football play and an improved winning percentage, but we’ll get to that in another post.)

They play in similar conferences, against roughly similar opposition – seemingly same caliber quarterbacks, receivers, and so forth.

Player A seems to effect a better overall coverage style. He is tighter off the snap, jams with far more natural leverage, is lightning quick in recovery even when juked by a head fake (which happens infrequently), is fluid – and while he can be outrun by some speedy receivers on fly routes, he makes up for this by excellent “natural” anticipation and a super quick start and acceleration.

A’s standing vertical leap isn’t so great. But watching him carefully suggests he does something critical to good pass defense, and very difficult to do for even better NFL caliber cornerbacks: He is extremely agile, and able to always get near maximum vertical air penetration from almost any angle, even while running and twisting to find the ball.

One can assume A has “good game” hands. Far too much is sometimes read into pass interception statistics, since often the sample size is far too low to make any real strong inferences. But our guy hasn’t dropped too many passes that have hit his hands, so his hands during games appear “soft.” (This is a key skill for DBs, to the extent it can be properly evaluated, It is also one of the most difficult, yet important, things to both teach, and require, in NFL caliber defensive backs if you want your team to dominate an opponent, or at least be Super Bowl bound. Even more so for receivers.)

Let’s switch to cornerback B:

His performance overall seems about the same. Maybe just a tad lower, but marginally, if even that. He seems to have a higher vertical leap, and also reasonably soft hands. Perhaps slower off the line, he seems faster overall, and is 3 inches taller than cornerback A.

Now let’s look at his measurables. Combine style. He’s 6′ 2″. His vertical is also 5″ higher than cornerback A.  Add in his 3 extra inches of height, and it gives him a total of 8″ over A. If his “wingspan,” or length from fingertip to fingertip with arms outstretched at 180 degrees and parallel to the ground is larger, this would give him additional reaching length. (Wingspans aren’t always correlated with height. I know this because I’m on the shorter side, but have an extremely wide reach, and very large hands.)

Now to the real beauty test for many players, including cornerbacks. The 40 yard dash. At 4.35 he’s very fast.

But is the 40 yard dash really that important? In terms of affecting draft positioning, it is, but it may not always be as good of an indicator as is presumed. (Jerry Rice, who some say is the greatest receiver to ever play the game, was repeatedly slow in his 40 times. Very slow.)

Sure, the Bengals’ Vontaze Burfict had other issues. But a big part of the reason he plummeted in the 2012 NFL draft was a poor 40 yard dash time. I wondered which team would take a shot at the possible steal of the draft.

The Bengals picked up Burfict – one of the stronger, if not one of the best, linebackers in the NFL – as a rookie free agent.  That is, he went undrafted. This was a huge mistake by other teams. And not just a “mistake in hindsight”:

The risk on Burfict was that he doesn’t work out – big deal, that happens with a lot of draft picks, particularly later ones. But Burfict had already been recognized as having 1st round talent. Turned out, he did, and has lived up to that in the NFL. (Unfortunately late last season he experienced season ending injury issues.)

Our cornerback A, in comparison to B’s fairly fast time, clocks in at a more leisurely 4:45 in the 40 yard dash. (This would put him around the middle to lower middle among cornerbacks in the 2014 draft, for example.)

It turns out cornerback B also did several more “reps” than A in the bench press – a flat bench press of 225 lbs, as many times as possible.

The rep bench press may not be as key for cornerbacks, but it may add to perception – well founded or not, that cornerback B is “disciplined” because he has hit the weight room. (Which on the flip side would tend to mask a lower natural strength than, say cornerback A, or for other positions, player A, who didn’t for any number of reasons, several of which don’t detract from his potential as a pro.) Or that he is “stronger,” which again may be incorrect. It’s a specific type of muscular strength, and also tied to direct practice of that particular lifting action and similar ones.

So the problem now comes with interpretation. We’ll look at this, and then integrate the first dilemma – real as opposed to perceived college play performance – with these same two players. In doing so, both key impediments to good draft evaluation should become more apparent.

Both cornerbacks are “good.” But cornerback B is a good bit taller. He extends that height possibility even more on jumps, with a more powerful vertical leap. Even more important perhaps, the vertical “tells us” that Cornerback B is more “explosive.” (The problem is it doesn’t necessarily tell us this. It tells us he may be.)  This explosion is key off the line, as well as possibly for fast shifting movement.

Both A and B played against similar opposition (which is often not the case, and is often far too underestimated), so we’re even somewhat comparing “apples to apples” (or players to players) here. Even better! (Or are we, as we’ll see in a moment,)

A is good, but B is “money.” He is taller, more explosive, “stronger,” and, drum roll, a good bit faster right in that “money” spot between being able to “okay” cover, and easily hang with the faster receivers in the NFL.

The following is not precise, but a complete hypothetical merely for the purpose of illustrating the two key points much more clearly. (And in more interesting fashion.)

Suppose looking just at film, both CBs looked like mid to late first rounders. But check out B! Those measurables put him significantly ahead of A, particularly with that fast speed, or 40 yard dash time. Measurables on the other hand, because of some okay, but not great, numbers, put A a little further behind than he’d otherwise be.

Draft day rolls around, and Team B trades away some draft picks to try and grab player B. All of this reflects a huge loss in value, since team B is not “lessening” their chances of getting a star, solid player, or even contributor with those other sacrificed picks; they are effectively making the chances zero with those picks they have now given up.

And so with the move, team B moves up from mid late in the first round, to early first round. And they “scoop” up player B.

Team A watches the board closely, trying to gauge both various team interest as well as player availability, while putting some feelers out yet playing it fairly close to the vest.

Team A targets two inside lineman and a linebacker who may fall because of position, as well as a smooth and rugged but not blazingly fast wide receiver, all of whom they think represent value for their late first round pick, and they think one to even both of the interior lineman, possibly even the wideout, may still be available early in the second round when they have another pick.

So team A hangs tight, and maybe the best player of the other four they were most aggressively targeting early, the linebacker (a position that used to get too little attention in the NFL draft relative to its team value, but that has changed in recent years), falls to them at 25. Bingo.

The other three of the four, along with our own cornerback A, are still on the board. And team A still has the 4th pick of the second round – 36 overall – due to a trade they made with a bad team (hence why the pick is the 4th of the round overall), who gave up way to much to target a player “they had to have,” the year before.

They love cornerback A, and think he’s a little bit underrated for our two reasons we’ll bring completely to light shortly, and may even have been better than the linebacker, who was a strong pick at 25.

But the wideout and the still available center (and maybe the guard), are both 1st round talents, and it’s not worth losing a draft pick just to “ensure” getting their CB by moving up a few spots. (That’s another mistake teams often repeatedly make: “Having” to get this or that player and paying an undue premium just to make sure they get him when they may anyway – thereby lessening the true value of the trade, since sometimes the trade did not “get them” that player in the draft, it merely upped their chances, a lesser value.)

They hang again, and watch with delight, seeing if someone will overpay for their 36th pick, since they know they’ll still get one of their 4 remaining guys, and if the best is still available, they can move back up for less. No one meets their modest asking price – albeit a little higher than normal, because they have their eye on CB “A”; and so if they are going to give up pick 36, they will ask for a little more than they normally would.

And voila, after a run at picks 33 and 35, the wideout and our center are gone, but the guard and our target CB are still available when our Good Old Commissioner (right now, still Goodell), calls up pick number 36. Bingo again. Cornerback A is off the board.

Fast forward 10 years. Who had the better career?

You can’t tell. And that’s the first problem with hard core draft moves that forget to sufficiently factor in the inherent variance factor.  On the facts we gave you, however, and completely forgetting the fact that one was a high first round draft pick which the team gave up some other picks to move up and take, and the other an early second rounder, which player is actually the better player?

On these facts it’s harder to tell. If all the facts were the same, but both players were a little worse – say late 2 to 4 round material without their measurables – the differential might be slightly more pronounced. The reason for this is that it’s harder and harder to play at the highest levels of the NFL relative to college, and making the transition from the college game to the higher levels of NFL play is a bigger jump. The difference is probably small, but in this case college performance not be quite as directly correlated, and some weaknesses – which measurables may or may not expose – might become more relevant.

However, this gets fairly speculative, and complex. The bottom line is that in general, on the facts we gave you, the two CBs are probably close to equal, and player A might be the better cornerback. It’s hard to say without more evaluation.

Then why did team B trade up? Why was the consensus consistent with (if perhaps not quite as enthusiastic as) team Bs? Was the consensus wrong??

For the most part, it probably was.  And this brings us to the first part of the two challenges. The integration of measurables into on field performance.

Simply put, when players A and B are evaluated based upon play, the role that their measurables plays in terms of effecting performance is often not isolated out, and removed; or not sufficiently done so.

Suppose player B looks better on the field. The reason he looks better might simply be because he is faster. So having a faster combine time might not be an additional benefit, or at least nearly as much of one as it is being perceived to be.  Alternatively, the reason player B looks as good as player A is because he is faster.

Without that speed, he may not look nearly as good on the field. And this may make up for a lack of inherent extra skill, that player A, on the other hand, has.

It’s legitimate. B is what he is. But his speed may be making up part of “what” he is as a player on the field. So getting ga ga over his 40 yard times, and disappointed over player A’s 40 yards times, while understandable, might be double counting a measurable which is already being inherently integrated into each player’s actual performance.

Of course it’s not this simple, as there are certain attributes that may translate better to NFL play, which is more difficult than college level play, and a little bit different.  There is also the even more qualitative, and tricky question of “potential.” Was B not superior to A on the field because of teachable or practice-able skills or inherent movement and kinesthetic patterns, flexibilities and strengths that A has already learned?

So the measurables can add value. But it is so easy and often natural to greatly overestimate their value, and confuse speculative judgment of “potential” with the actual fact or high probability of it, that a lot of teams would almost be better served to all but ignore them altogether. At least until a more robust way of integrating them into actual on field performance is practiced.

But again, on these facts, player B could still be a little better. He may have had less relevant football experience, he may lack some of those skills that he has the potential to learn – hard as this is to figure out accurately – etc.  There’s also the second key factor we will get to in a moment, which could also cut either way. We don’t know. But on these facts as given, the perception of player B as a better cornerback than A, may be pushing it.

Fast forward to 10 years later.  Cornerback A had a fantastic career, Cornerback B was a solid player – not a draft “bust” who sometimes started, then faded to a strong nickel position.

How was A the solidly stronger player, when the consensus had B as the top corner or second corner of the draft? Pure variance? That same variance that teams inevitably underestimate when they target players and and make many draft related trades?

Yes. And of course, as we already looked at, the likely role of the measurables – though that doesn’t necessarily explain why A is better, and again, it is possible that some of the B’s “raw” talent and size and speed, if they are otherwise close could translate better to the NFL.

Looking more Deeply at Both Player’s Real versus Perceived On Field Performance

But there’s also something more, which can be further integrated before the draft, and in this case, it may explain why A was the better cornerback in terms of his ability and likely NFL career, coming out of the draft.

Unlike measurables, which of course need to be looked at more closely yet given less overall weight or value in the total “grading” process, this factor needs to be looked at a lot more closely, and along with overall on field performance, given a lot more value. In other words, decrease the weight put on measurables, and increase the weight put on this next factor, and your team will make better draft decisions overall just on this alone.

The two players played in similar conferences and against somewhat similar teams. But when in the field, were their experiences similar enough, or is their something to distinguish them further?

Both seem to have good hands. (A good thing for all football players, by the way, and under practiced and under insisted upon.) Let’s look closely at an often wildly misleading and therefore very tricky statistic. Interceptions:

Did B have 4 balls bounce off his chest, but catch 18 others in three years? Okay, bad hands, but how did he manage to get into position so many times! It’s still a small number – some of the picks could have been part of the whole “that’s the way the ball bounces” (or in this instance, “flies”) in football; but the number is getting large enough to start to give some solid value to it.

Did B take a lot of “chances” and have a lower rating than he would because he is burned a lot?

Strangely, if it’s only a little bit (in other words, most of his chances were well decided, with some protection against the catch in case he missed the pick) that might speak to some upside if B is willing to undergo what it takes to learn to catch a football. (Note to NFL teams. Catching a football, whether you are going to then be tackled or not, is not a difficult act. Teams don’t know how to effectively practice it to make it like second nature.)  Then very slightly cutting back on, or slightly tweaking and working with his route jumping may even further lessen the times he is flat out burned, while still leaving him with a lot of picks due to immeasurable on field instinctive and movement skills, and now better hands.

But let’s look at our player B. Our player B had 14 picks, and like A decent seeming hands. Player A had 9 picks. This high pick number is part of the assessment on B’s overall potential. But pull out the tape, and examine carefully. 5 of B’s 14 were against backups after injury, who were themselves having bad games, and were easy picks that happened to come his way.  4 more were at the ends of games on 3rd and 4th downs by the losing team where the QB simply had to “take a shot.”

On the other hand, only 1 of such picks fell into player A’s hands, while 7 were actually tough catches, and the 8th was an easier catch after a sick last second cut inside on a soft slant.

Player A also never faced a backup QB. And while the offenses “seemed the same,” almost every game he played happened to be against an opposing starting offensive line, without injury; he played several games against the top passing offenses with his team’s two best pass rushers sidelined, and, evaluating between the two college careers, he faced slightly better offensive lines, along with better quarterbacks, than B faced.

Overall the receivers A matched up with, often in single coverage and rarely with help, were probably equal in ability or maybe better than those B covered. (This is also hard to evaluate, because of the many factors that go into being a “successful” wide receiver that have to do with the rest of that wide receiver’s team and offense in addition to the player’s own ability and performance.)

It turns out A also played with a strained hamstring for 6 games his senior year, that was heavily under reported. It pained him to run, but he didn’t flinch, and sat in hot tubs and stretched lightly all week long to try and get it ready and heal more quickly.

Remarkably (and a bit recklessly, since a little time off for at least deeper initial rest probably would have bee more productively) his extra effort off the field allowed him to recover anyway. A didn’t lift because he didn’t want to bulk up, and thought he’d see how to best increase lower arm and effective “reach” and quick movement strength when he got to the NFL and saw what he was up against.

In this case the bench rep press was no indication of off field “commitment” at all, and A was an instinctively smart player who knew how to grow his skills, worked as hard as player B off the field, and was committed to the game.

Looking more deeply at the film, we see that the percentage of drops against player B – random drops, bad drops, drops that that should not have been dropped, were much higher in his senior  or last, year – the most draft defining season for a college player. Had most of those not throws not been dropped, B’s overall ability may have been considered lower.

On the flip side, while A did a good job sometimes blanking them, he was often up against the other team’s best receiver, and a star for that team, and there were also several good catches against him in really tight coverage, where most receivers would have dropped the ball. These also artificially deflated A’s performance perception, whereas a bunch of drops against B, may have simultaneously inflated his.

The foregoing “deeper look” into both players on field performances is an oversimplification of course and slightly exaggerated, and may be too obvious example; but it is a good example of the more complex, deeper process that must be analytically evaluated and compared, and, most difficult of all, adequately adjusted for.

This is hard to do, since, for example, in player evaluation we’re apt to go “sure he had a great QB throwing him the ball, but look at how good he is!” simply because we see him catching the ball a lot, including some good catches (just as other receivers with lesser opportunities may also make, just with less opportunity to do so) – when other receivers that are not considered as good, might in fact be just as good.

Thus, in addition to measurables – the first second factor which needs to be dialed back – the second factor, is one that needs to be dialed up, and it is that on field performance has to include robust evaluation of the specific players and under what conditions our guy went up against, and even who his opponents in turn went up against, and how they fared, in order to get a better assessment of their real skills so that we can in turn get an even better assessment of our guy’s skills.

Summary

The bottom line for these two key underestimated and insufficiently integrated factors is, of course two fold.

First, measurables, carefully considered, can offer value to the draft evaluation process. But very often those measurables are to some degree – and sometimes to a very large degree – already factored into the players on field performance.

And but for those measurables – which are then being used to “boost up” a player’s draft status on top of his on-field performance – said on-field performance, in turn, would actually have been much weaker to begin with. And the player would have thus been evaluated even lower based upon on field performance, excluding strong “extra value” measurables, to begin with, if there was a way to do that.

But there isn’t. We don’t go “great play, but he’s not that good, look at how fast he is.” But we are apt, after a combine 40 yard dash test, to later go “wow, look at how fast he is, he’s better than we thought!” But he may not be.

The second factor is the importance of looking at football first and always as a team game, even for somewhat “island” positions such as cornerback (and even more so for many others). And in doing so, carefully evaluating the quality and performance of the actual and specific opposition faced, right down to the specific players involved and what factors might have enhanced or lessened their overall ability or projected ability at such times as our guy faced them.

Even when this is done, it is human nature to discount this far more than is warranted, based upon what we “see.” It is what we don’t directly see, that is often the more important. Better drafting decisions, requires it.