Friday, May 10, 2013

Luck vs Skill - Is anyone good at picking football players?

A couple of weeks ago I took a look at the randomness in NFL Draft results, prompted by articles from Brian Burke and Chase Stuart on and, respectively. I found some slim evidence for skill in drafting – the streaks of drafts with outperformance were slightly longer than expected. Interestingly there was much stronger evidence that some teams are not good at drafting, but you can read that post to get the details there.
The year-over-year data, however, has a major issue that I mentioned in the post: it’s not the same people making the decisions. Looking at the picks made by the Browns in 2012 and those made in 2013 provides no insight because the regime had turned over in the interim. Above and beyond that, the competitors do not behave similarly year to year so the decisions of each team – the variable we are trying to evaluate – are mixed in with changes in the way that other teams decide.

In response to these problems it seems like a study of the picks within a single year would be the place to look. The premise is that any team that has made a good pick should be more likely to make a good pick with their selection. If the initial pick reveals anything about their skill level, the subsequent success rate will exceed that of teams with a miss on their previous selection.

Using my data set of draft performance in the salary cap era[1] and my analysis of what a draft pick is really worth, we can quickly look at which picks were successful by identifying the ones that delivered more than the expected value. The percentage of picks that are successful varies widely by round due to scouting focus and the binary outcomes in later rounds[2]. If a player makes it and stays in the league for a couple seasons, they probably generate enough value to be a success because the expected return is so low in later rounds. Most of the picks in these rounds wash out: 269 of the 569 7th round picks play 1 season or less, and a grand total of 0 of them delivered value in excess of their expectation.


The first way of analyzing picks will borrow from the methodology used in my first post on this subject (which incidentally borrowed from a study of mutual fund managers’ success highlighted by Michael Maubbousin in his book The Success Equation). Each pick has an expected success rate based on the average success rate in the sample – I use the success rate of a round which is somewhat arbitrary but reflects the way that team leadership could realistically be expected to approach the draft.

With this expected rate of success for each round I ran simulations of the real drafts 1,000 times so that the simulation set would have the same profile as the data set in terms of sequencing (e.g., the Browns drafted 6th, 70th and so on in 2013 and the simulation would have them do the same). To demonstrate, here is what 20 simulations of the Baltimore’s 2006 draft looks like with a 1 being a good pick and a 0 being a bad one.

Applied to the full data set, this methodology shows us how many streaks of a given length we should expect to see given the actual order of selection and average success rate for each round.

The results show no evidence for skill in drafting. The number of streaks we actually find in the data is well within the range of the number of streaks in the simulation for both streaks of good picks and streaks of bad picks.

One on one 

But wait, you say, isn’t it possible that since teams might spend a much higher amount of time evaluating early picks that all this streak stuff misses because the relevant streaks are only 2 picks long? Excellent question! To answer it let’s take a look at the individual picks by sequence and by round.  

There is not a lot to see here. The P-Values are all over the place and while a few are significant, looking at this many data points we are bound to find a few that are significant by chance. I have a hard time believing that the outcome of the prior pick is significantly correlated in the case of a team’s 8th pick even after being unrelated for the first 7.
On those extra picks in the first round we can certainly put together a story that includes a general manager freed of the pressure to conform that they felt with the first selection who boldly selects the underrated player they secretly knew all along would be a star, but it’s a very small sample size and the simulation standard deviations reveal them to be well within the range of the expected results given the average success rate.

A paradox

I don’t want to use this analysis to suggest that there is no skill in drafting – I don’t think 31 fans off the street would fare particularly well against 1 real GM with a full scouting organization – but there doesn’t appear to be excess skill. That 1 GM with a full scouting organization is competing against 31 other GMs with similar resources and precious few proprietary insights. All of them pore over the same game tape and combine stats save a few private workouts with some prospects.

To go back to Maubbousin’s book, what we have here is the paradox of skill in practice. As teams are very skillful at drafting there is less excess skill for any one team and the outcome is dominated by luck. Based on this data, there may be even less skill involved success or failure of NFL drafts than there is in success or failure of mutual funds.

Note - I went a step further and looked only at those picks made after a team traded up. See the analysis here.

[1] Looking at the drafts from 1994 through 2006 with a total of 3247 players
[2] Hat tip to Football Perspective commenter Eric Barry for proposing this would be the case due to scouting focus on early picks


  1. I ran some numbers based on the previously posted draft grade data from 2000-2007. No teams are better than average drafters, though Oakland and Detroit, over that time same time, were statistically worse than average at p<.01 using a students t test.

    Unfortunately, this data doesn’t include 96 and 97 where the Ravens had two of the better drafts ever.

    There is massive variance from year to year within a team, which lends credence to the notion that luck probably has a lot to do with it. Then again, I can’t help but wonder if there is more to do with the success discrepancy than only luck can account for, especially when you have teams like Baltimore, New York (Giants) and Pitt that consistently produce solid draft classes with a small variance in value (better than average overall skill at drafting) and are perennial contenders, as opposed to say Dallas or New Orleans, which had one great class (luck) and 7 average ones (skill level) producing a massive variance in draft value, and are typically fluky contenders. I don’t think you are consistently lucky for years on end, as the good teams seem to be, or as consistently unlucky as the bad teams look.

  2. ERych - Based on the data set posted by Chase Stuart (the basis for my first post on the topic: which includes 1970 to 2007, I did not find evidence of more positive streaks than randomness would suggest. You make a good point, though, that bunched good drafts with a bad one thrown in would not demonstrate an abnormal number of long streaks but would support the idea that some teams are better overall.

    The complication for me is applying a consistent value to who is picking - even in an organization without personnel change there might be a different person with final say from year to year.

  3. The problem with statistics is always in the application. For instance, 8 years of data tells us that the Ravens have drafted better than the Raiders over that time frame. Does that mean the Ravens drafted better every year than the Raiders did? Or that the Raiders can't have one good draft out of 8? Or that the Ravens cant have one or two bad drafts themselves? No. What it tells us is that the Ravens have produced on average, better valued drafts than the Raiders, and that if I had to pick a front office for the next 8 years, I'd rather have the Ravens, because history says the Ravens out produce the Raiders on draft day.

    It doesn't take multiple good draft picks in a row to produce a good draft, or a string of good drafts, just like you don't have to win every hand of poker to come out ahead. You just have to win more money than you lose, and if you do, the house can argue the numbers all it wants as you drive to the bank, the same way Oakland could claim the Ravens were just lucky for 8 years, and they had a run of bad luck.