Replacement Officials: They are that bad

Replacement officials in the NFL have been mildly controversial lately. From players to coaches to numerous columnists, it appears that the call for the real refs is universal.

Several of my usual reads have already analyzed various aspects of the situation. At advancednflstats.com, Brian Burke took a look at home field advantage and concluded that the numbers for this season are not conclusively different from the long-term trend. Bill Barnwell comes at it from more of an “integrity of the game” approach that a great number of columnists are favoring.

Are they really that bad?

Year	Avg Penalties per Week (1-3)
2005	237.3
2003	233.0
2004	218.33
2012	218.30
2010	208.7
2011	207.7
2002	203.0
2009	200.3
2008	191.3
2006	188.7
2007	182.0

Stats-wise, any time I see this kind of consensus opinion I want to look into it and see what’s up. Through three weeks, the replacement refs are averaging 218 penalty calls per week. This puts them comfortably in range of the past ten seasons when compared against the first three weeks of each. When compared against the full-season numbers, 2012 looks terrible with only 2004 and 2005 above 200. But this is because the replacements have so far participated only in 16 game weeks while full seasons have several weeks with 13 or 14 games to accommodate byes. Penalties per week also tend to go down slightly throughout the season even accounting for byes.

So if they aren’t calling more penalties, what is the problem? The 2012 season is seeing more penalties than 2011, about 0.5 extra calls per game, but that doesn’t seem like enough to cause this outcry.

The real issue is in the composition of the penalties. The table below shows penalty averages in 2012, from 2002 to 2011 and just in 2011 to highlight the different composition of calls this year.

 

Penalty	2012	2002-11	2011
Offensive Holding	43.7	37.3	36.7
False Start	43.3	46.7	42.3
Defensive Pass Interference	20	12.7	12.7
Unnecessary Roughness	12	9.0	14.3
Defensive Offside	10.3	13.1	12.0
Defensive Holding	9.7	6.7	6.7
Delay of Game	9	9.4	8.3
Illegal Block	8.3	8.7	9.3
Personal Foul	7.7	3.4	1.7
Roughing the Passer	6.7	6.4	7.7

In a typical season, the most common penalty called is a False Start. From 2002 to 2011, the False Start averaged 25% more calls than the number two penalty, Offensive Holding. In 2011 the numbers were slightly closer with 15% more False Starts than holds called. So far this year holding has been called more frequently than False Starts.

The numbers only get worse from there. The three penalties with significant divergence from prior numbers, besides Offensive Holding, are Defensive Pass Interference, Defensive Holding and Personal Foul. These three are huge in the flow of the game. Defensive Pass Interference has increased by 60% over the 2002-11 average, Defensive Holding has increased by 44% and Personal Fouls have more than doubled. If you want to see why the calls “feel” wrong, this is it.

This disparity should also come through in the average yards penalized per week, which takes into account the composition of the calls. The typical weekly penalty yards through 3 weeks are 1,699 with a standard deviation of 168. If we look at just the last five years the numbers are 1,606 and 133. In 2012 the average weekly penalty yards are 1,898. This puts 2012 just inside 1.2 standard deviations from the ten year average (significantly different at 75% confidence), but nearly 2.2 standard deviations from the five year average (significantly different at 97% confidence).

The bottom line is that these officials are demonstrably worse than the real refs, even if that is only visible a level or two below the top line penalties per game number.

Who blinks?

All the numbers in the world aren’t going to talk the NFL – specifically the owners – out of their bargaining position. The referees look similarly dug-in on their side. The details are obviously more complicated, but the NFL is looking to move from defined benefit to defined contribution pension, create some backup referees for any in-season changes necessitated by poor performance – currently only possible after the season – and add a full time employee at each position (line judge, referee, umpire etc.). I don’t have a particular side in this, but a bit of advice to some of the louder voices:

For those who say “it’s only 0.33%” of the NFL’s revenue (some sources say as little as 1/20^th of one percent), just remember that pursuing this argument implicitly says they’re worth anything they demand unless you’re willing to specify an appropriate percentage. Maybe the appropriate percentage is actually less than the current state.

For those who say demand for the NFL is robust and is not at risk from this dispute, just remember that things tend to be robust until they’re not. Also, seriously, it’s only 1/20^th of one percent (that’s 0.05%) separating the sides in terms of the NFL’s $9 billion annual revenue.