Thursday, May 31, 2012

Power rankings make my head hurt


ESPN.com produces some excellent analysis. Several of the writers working there can slice and dice data in ways that reveal thought-provoking, insightful conclusions. I have a weakness, however, for those delightfully qualitative pieces known as power rankings.

Even though I know that I should know better, I read them every week. I literally cannot stop myself from clicking on that link (figuratively, of course – I could quit if I really wanted to). Imagine, then, just how little control I had when I saw the NFL Future Power Rankings link on the lower right side of my screen. These rankings purported to give a quantitative look at the prospects of each team in the 2015 season.

I started to suspect something might be amiss when I looked briefly at the methodology. The methodology was to poll some of ESPN’s analysts and produce a weighted average of the results. There were instructions given for each category that seemed to make sense. Don’t consider players over 27 except at QB; look at the 2012 draft class, the team’s picks in the 2013-15 drafts and their track record in the draft.

In all, the five categories included Roster (ex-QBs), QBs, Draft, Front Office and Coaching Staff. The 4 analysts – Trent Dilfer, Mel Kiper (he of this track record), Gary Horton and Matt Williamson – voted 1 to 10 for each category and that was the extent of it.

The results of the survey looked all too familiar. There was Green Bay at the top, followed closely by New England, the New York Giants, San Francisco, Pittsburgh and Detroit. Not until seventh was there a team, Philadelphia, that failed to make the playoffs. Three of the four conference championship game participants made the top four. It did not appear to be significantly different from simply ordering teams based on their 2011 record. I decided to take a look at whether current year results are really that highly correlated with results in four years?