A couple of weeks ago I took a look at the randomness in NFL Draft results, prompted by articles from Brian Burke and Chase Stuart on advancednflstats.com and footballperspective.com, respectively. I found some slim evidence for skill in drafting – the streaks of drafts with outperformance were slightly longer than expected. Interestingly there was much stronger evidence that some teams are not good at drafting, but you can read that post to get the details there.
The year-over-year data, however, has a major issue that I mentioned in the post: it’s not the same people making the decisions. Looking at the picks made by the Browns in 2012 and those made in 2013 provides no insight because the regime had turned over in the interim. Above and beyond that, the competitors do not behave similarly year to year so the decisions of each team – the variable we are trying to evaluate – are mixed in with changes in the way that other teams decide.
In response to these problems it seems like a study of the picks within a single year would be the place to look. The premise is that any team that has made a good pick should be more likely to make a good pick with their selection. If the initial pick reveals anything about their skill level, the subsequent success rate will exceed that of teams with a miss on their previous selection.
Using my data set of draft performance in the salary cap era[1] and my analysis of what a draft pick is really worth, we can quickly look at which picks were successful by identifying the ones that delivered more than the expected value. The percentage of picks that are successful varies widely by round due to scouting focus and the binary outcomes in later rounds[2]. If a player makes it and stays in the league for a couple seasons, they probably generate enough value to be a success because the expected return is so low in later rounds. Most of the picks in these rounds wash out: 269 of the 569 7th round picks play 1 season or less, and a grand total of 0 of them delivered value in excess of their expectation.
Note - I've gone back to this issue a couple times since this post and looked at whether there is any evidence for skill in sequential picks in a single draft and in picks made after trades up. Check them out via the embedded links.
Both Brian Burke and Chase Stuart recently wrote on the phenomenon of drafting in the NFL and the randomness that goes with it. Chase in particular wrote about the question of whether some teams are better than others at drafting.
At the recommendation of my friend Sarah, I recently read Michael Mauboussin's book The Success Equation: Untangling Skill and Luck in Business, Sports and Investing. One of the many interesting nuggets in the book was that fund managers demonstrably exceed the number of multi-year "streaks" of outperformance we would expect to see if performance were truly random (identified in a paper by Mauboussin's son).
Based on the data posted by Chase here, I built a shoddy, homemade version of a Monte Carlo simulator and ran 2000 versions of the actual teams/seasons. With a few small changes (the '99-present Browns are a sad, weak imitation of the pre-95 version and I won't have them associated with each other), I ran the simulations with the actual teams and years played. This ensured that we maintain equivalency (e.g., the 2003 Houston Texans can have a maximum streak of 2 because it's only their 2nd season and so on) and it was a fun chance for me to hit F9 2000 times because the iterations were not working.
The results are below but I found two parts very interesting.
First, the outperformance was never significant at a 90% level outside of the first year, meaning there are fewer streaks started than would be expected. These extra seasons of outperformance show up in the form of slightly longer streaks (actuals average 3.96 years while simulated streaks average 3.76).
Second, there are some terrible teams in the NFL. The underperformance is significant at the 90% level from seasons 6 through 13 - far more teams are bad at drafting longer than we would expect by chance alone. For those interested, these teams are the Falcons (season 10 in 1990), Colts (1984), Cowboys (1987), Raiders (2007) and Chargers (2000).
The next level analysis here is to segment by team management so that different eras are recognized as discrete units rather than part of a larger whole. We should really see some differentiation there as "good" GMs stay employed and rack up streaks.
