In general it is a pretty good assumption that adding
information to a decision making process improves the outcome. In specific
instances, however, the added information can be extremely disruptive. With
respect to the NFL Scouting Combine, does it help? It depends…
The method
While we do not have visibility into teams’ draft boards
before and after the combine, we do have access to one consistent record of
pre- and post-combine sentiment from a consistent (and remarkably
well-coiffed) source: Melvin A. Kiper, Jr.
By pulling several years of draft predictions (2004 to
2008) and evaluating the change in quality of predictions on either side of the
combine, we can evaluate whether they improved. For each player, the absolute
difference between the value they have delivered and the value of the player
picked in that position will be the quality of the prediction: the lower the
better. If, before the combine in 2004, Kiper projected Larry Fitzgerald to be the
1st pick, then the quality of that pick would be the value provided
by Philip Rivers as the actual most valuable draftee (64% as ranked by Career
Salary Cap Value) less the value provided by Fitzgerald (52%). The average
of all those individual values is the aggregate quality of the prediction[1].
A prediction with 0 error will have all players ranked in order of their actual
career value to date.
Remarkably (or maybe not remarkably, he’s been doing this
for a long time), Kiper shows an improvement at the aggregate level from before
the combine to after. Across 152 individual players the average improvement in
the prediction per player is 0.03% of the salary cap closer to the actual value
of the players. Of course this 0.03% means that the average prediction improved
by $38,000 of football value at the current cap number. The real action is
looking at the distribution of increased (or decreased) accuracy for each
position.