Monday, May 5, 2014

Looking for evidence of better drafting

Chase Stuart put up a post last week examining whether (and why) teams had gotten any better at drafting. He looked at the proportion of value delivered by top picks relative to all picks in the draft (i.e., can teams identify the value in each draft) and at the correlation between draft position and career value from year to year (i.e., does where a player was picked predict career value).

Given that Chase examined the relative ability of teams to pull value early in the draft, I wanted to see if drafted players in general changed over time.

To see if I could tease out any improvement in drafting I looked through my draft database from 1994 to 2012 (starting with the implementation of the salary cap).

As you can see in the charts above, teams have been getting substantially more production out of players in their first season without much change in the number of games played (these numbers are adjusted for expansion in 1995, 1999 and 2002). The implication is that they are getting better performance out of drafted rookies.

Looking at the next three seasons – all of which would be under team control – we’ll be able to see if the trend holds up.

With these data points the interpretation of the first chart changes. Rather than showing players who are improving, they’re simply coming out of college more fully formed. The trends in games played and production are that there are no trends.

Teams may get a bump in early performance, but the aggregate performance of a class of drafted players looks like it tops out in the same range as it has since the 90s. The numbers are remarkably steady through the 20 year period.

As far as the specific question posed by Chase: Why aren’t teams better at drafting? There may simply be a limit to how much teams can tell about a player’s underlying talent and likelihood to hit their potential. With top players going to the same set of trainers and draft prep specialists, it’s no wonder it hasn’t gotten easier to tell the busts from the stars.

1 comment:

  1. "Rather than showing players who are improving, they’re simply coming out of college more fully formed"

    I thought the point was to show improvement in team's scouting/drafting abilities?

    I think the only thing this really tells us is that AV works, as adjusted for era, that player potential has been accurately adjusted for over the past 20 years using the AV formula.

    You get very little production out of late round picks, and the same pools exist (smoothed over the years) at every draft, so it makes sense that max production hasn't changed, as the elite players get drafted regardless of team scouting every year, and these are the players really making an impact in average team AV.

    Its kind of hard for me to wrap my head around, but I don't think we have the data available to prove out something like this (at least not using AV as your independent variable). Your approach I think really could only tell us if players are better now than they were years earlier at certain points in their careers, which is more about development in college and the pros than it is about drafting ability, as I described in my previous paragraph.

    You could probably only compare teams to league average, and look for better teams (turnover in the front office compounds this problem). If the NFL as a whole is spending all this extra time and effort, nobody ever actually gets better, because improvements are negated across the board.

    Nice post.