The 'D' Stands For ...

... Doesn'tmakeadifference.

As much maligned as Jeff Lurie's "Things were terrible; I'm not changing anything" remarks were a couple weeks ago, this is one time where he's in close alignment with at least a sizable minority of the paying customers.

Ask any Eagles fan if promoting Juan Castillo to DC was the right move last year and it's unlikely you'll hear the word 'yes.'  And yet somehow one-quarter of our brethren still think Castillo deserves a second chance!

I don't really take this number at face value.  Too many of the explanations I've heard for why we should keep Castillo start by blaming Andy Reid for hiring him in the first place, as if what matters here is ensuring Reid pays for his long litany of (supposed) sins and not finding the best possible person to coach the Eagles' defense next year.

This is, of course, moronic, but also the logical next step in the long line of "If we just get rid of him, everything will be better" arguments we've heard for about the last decade.

At any rate, I'm not really here to talk about Juan Castillo (which avoids rendering the last paragaph intolerably ironic), but rather a subject near and dear to my heart:  DVOA, and the misapplications thereof.

Let's start by stipulating that DVOA does all the things a good stat should do.  It's descriptive and predictive, gives us a common metric for measuring teams against each other, and boy does it help resolve a lot of arguments.

To the extent DVOA has problems, it's not because of anything the FO guys do.  Look at last week's FO NFC Championship game preview. There are many words and arguments in that thing, but not once will you find the sentence:

"And of course San Francisco will win because it has a higher DVOA."

Such a statement would be absurd on its face (and would also make people wonder why they're paying for FO's "advanced picks" each week when they could just look up two numbers in a freely available table and get the same answers.

So let's emphatically state that this isn't an FO problem.

But boy, is FO starting to cause some problems. And the biggest, I think, stems from a near universal misunderstanding of what the "D" in DVOA means.  Start with their definition:

Once we have all our adjustments, we can find the difference between this player's success and the expected success of an average running back in the same situation (or between this defense and the average defense in the same situation, etc.). Add up every play by a certain team or player, divide by the total baseline for success in all those situations, and you get VOA, or Value Over Average.Of course, the biggest variable in football is the fact that each team plays a different schedule.

By adjusting each play based on the defense's average success in stopping that type of play over the course of a season, we get DVOA, or Defense-adjusted Value Over Average. Rushing and passing plays are adjusted based on down and location on the field; receiving plays are also adjusted based on how the defense performs against passes to running backs, tight ends, and wide receivers. Defenses are adjusted based on the average success of the offenses they are facing. (Yes, this is still called DVOA, for the sake of simplicity.)

Sounds pretty simple, right?  VOA is a measure of how well a team has played and DVOA corrects for opponents, so you can throw out any concerns about strength of schedule and say things like "DVOA proves the Eagles have the league's twelfth-best defense, even if you claim they didn't play very many teams that were any good."

Except you can't.  DVOA says nothing of the kind, as we can best demonstrate with a couple of charts.  Here's the first one, plotting this season's DVOAs against VOAs:

Yes, that looks pretty much exactly the way two data series with a correlation coefficient of .985 should look when plotted next to each other.  Here's schedule strength (based on DVOA) vs. DVOA:

That one's messier, but there's a clearly identifiable down trend (correlation = -.660).  If the opponent adjustments actually worked the way people thought they did -- and which I thought they did until not that long ago -- the correlation here would be much closer to zero.*  

Again, none of this is meant as a criticism of Football Outsiders.  They're trying to create the most predictively successful stats possible and they appear to have settled on minimal opponent adjustments as the way to do that.

It's easy to see why making bigger adjustments would be tough.  NFL teams play better at different times for a wide variety of reasons (injuries, scheme adjustments, new coaches, etc.).  Given the small sample sizes involved, it's practically impossible to start saying things like, "Well, Trent Cole was out that week, so we should bump the ASR, but oh wait, so was their LG, so we need to factor that in too."

What this means, though, is that it's time to stop hand-waving away posts like this one from Brian Solomon with the explanation "Oh, DVOA takes care of all that stuff."  No, it doesn't, and if you still don't believe that, I've got some Stephen McGee futures contracts to sell you.

---
* Note:  The strength of schedule effect is weaker in previous seasons.  The DVOA/VOA correlation stays in the .97-.98 range, however.