Lewis Pollis, a recently graduated senior at Brown
University and past/future intern for several MLB teams, put together an
analysis of the value
of a general manager in Major League Baseball for his senior thesis in
economics. He also had a summary
version posted to Regressing, Deadspin’s analytics-oriented sub-site.
The paper estimates the value of the top-performing GMs
to be tens of millions of dollars higher than that of middling GMs, who are
likewise tens of millions of dollars more valuable than the worst performers. From
there the paper links this value differential to the current narrow band of
salaries and advances the notion that “the best baseball operations employees
are paid substantially less than they are worth to their teams.”[1]
What is being measured in this research is how well teams
perform. This performance is measured across a narrow set of activities
(signing free agents, making trades) and attributed to the specific GMs who
conducted the activity with some caveats about what can and can't be included as well as how it is measured. I am bought-in on this approach. These are hard things
to measure and this seems to be a good way to get at it.
The outcome of this performance measurement is a
distribution of value generated from best to worst and the conclusions noted
above – the value that a good GM provides over a bad GM – flow from this distribution. This is where I have a couple of issues.
A couple of issues
For one thing, Pollis notes that the specific GM is not
an accurate predictor of how a trade or signing will work out[2].
If a GM is not a predictor – or more accurately if making good picks in the
past doesn’t make them any more likely in the future – then the value
difference may be coming from luck rather than skill.
In the posts I did looking at the NFL draft I found no evidence for excess skill in selecting players. In the year-over-year data there were a few more
streaks of bad selections than a purely random model would suggest, but the
streaks of good picks were well within the expectations from
the random model.
Another point offered in support of the model is that
“subjectively speaking, the individual rankings seem more closely aligned with
how well the GMs’ teams performed than with outsiders’ views of their
decision-making processes.”[3]
I would suggest that this is somewhat worrisome in that it implies the model
may simply be a more complicated way of measuring success for teams – the model
is drawn from wins above replacement, wins above replacement are highly
correlated with team performance, team performance is what tends to drive GM
reputations. Now I’m not sure there is a good way around this but I would be
interested in any counter-intuitive results – maybe some GM has had bad injury
luck in the draft but free agent signings and trades show a skilled evaluator.
Finally, the paper notes that the correlation between
GMs’ measured ability in trades and measured ability in free agent signings is
effectively 0[4].
If the model were measuring the combination of skill in identifying good
players as well as skill in paying them salaries advantageous to the team – and
this is what I understand the model is attempting to measure – we should expect
those skills to be common across trades and free agent signings. I can’t think
of a skill-based explanation for why a GM would excel at one and not the other.
I can think of a non-skill-based explanation: luck.
Conclusions
I enjoyed this paper, and I think this is a great start at separating good
from bad in GM performance. What I would like to see is more effort put into
demonstrating that good or bad performance relates to the specific GM.This is far easier said than done and might require a lot more observations in the data set.
In the absence of additional data, the fact that skill across trades and free agent signings
is not correlated leads me to suspect the answer is that GMs who measure highly
were unusually lucky in these activities while those who measure poorly are
unusually unlucky. This fits squarely into the paradox of skill – the importance of luck in determining outcomes rises as the overall group
becomes more skilled.
In baseball, as in football and investments, the
decision-makers tend to be very smart people and the organizations have
developed sophisticated infrastructure to evaluate talent. The result of this
is that individual teams are not likely to be significantly more skillful in
their evaluation than others and the outcome will increasingly defined by luck as the skill level rises. See here for a fuller explanation.
My biggest disappointment with all of this is that the
next iteration is unlikely to be available for public consumption with Pollis now
working in-house for the Cincinnati Reds. I hope he still manages to publish occasionally, and I wish him and the Reds good luck.