I left a small cliffhanger in my last post. After a long week I finally had a chance to read through the Adams paper about estimating the value of “going for it” on 4th down. I admit I was a little bit let down. As a reminder – the question is what action a football team should take on fourth down. Failure to gain the necessary yards means the ball is turned over to the opposing side, kicking turns over the ball but with better field position, and making the first down allows the drive to continue, potentially leading to more points. The conclusion of the Romer paper was that coaches are too conservative and kick the ball away in situations where they should go for it instead.
Adams hits the nail on the head by asserting that the results of the Romer paper just do not pass the “smell test”. It’s nuts to suggest that it’s a good idea to go for it on 4th and 4 on your own 25 yard line. But that leaves us only with more questions – is the conclusion of the Romer paper still valid, even if overstated? Can we identify a flaw in the reasoning? Is there a better way to model the problem?
Adams first suggestion for improving the model is to include more historical data. Adams and Romer both claim it’s hard to come up with a good model for the “going for it” problem because teams seldom go for it on fourth down in practice – data is hard to come by. Romer and Adams both use game data from the 1998 – 2000 seasons, but Adams uses data from the entire game, not just the first quarter. But why not include more recent data? [The Adams paper was written in ’06, so he could have doubled the data set. We have a couple more seasons-worth of data now.] So I’m not sure I even buy the premise that data is lacking.
Adams’ second approach is to use Madden ’07 to simulate 4th down situations. I initially thought this was a really cool idea, and it kind of is, and then I remembered something I once read. Madden himself asked the designers at EA to make 4th downs more difficult to convert! You cannot find a better example of Galbraith’s notion of “conventional wisdom” in action. So as far as I am concerned, you have to throw out the middle section of the paper. Madden is not a simulation: it is pretending to be a simulation. It wants to make you feel like you are experiencing real NFL football. But the problem is that we as players do not make decisions the way that GMs, coaches, and players do. Our motivations are completely different, and there are no real consequences for our actions (other than bragging rights over your roommate). My GM will not fire me if I go for it on 4th and 5 on my own 25. Thus the game must be tuned to correct for this, otherwise you will get Tecmo-like gameplay.
The last section proposes a game-theoretic approach. Adams introduces a zero-sum game with the offense and defense as opponents. The offense and defense both have the choice of choosing a pass- or run-oriented strategy. The payoffs depend on their choices. Adams points out that this is a “simplified version of reality.” (It’s very close to the original Tecmo Bowl – two choices instead of four.) He uses this approach primarily to make the point that it is not a good idea (as Romer proposes) to use third down data to model fourth down choices, because the payoffs change enough to matter. It is an interesting line of argument for the claim that Romer’s conclusions are overstated, but it does not provide insight into how to better model the problem.
Anyway, in the course of poking around the web I came across the ZEUS Football simulation engine. It is frequently referenced in the NYTimes “5th down” blog. For example, here is an interesting discussion about taking an intentional safety late in the game. (I won’t bother to explain what that means, because if you have made it this far, you clearly already know what I am talking about.)
All the questions I raised at the beginning of this post are probably best answered by a simulation engine. Which reminds me – did I mention that Solver Foundation is adding stochastic capabilities for our version 2?