The Jets, my personal favorite team, had gone 12-4 in the 1998 season, going all the way to the AFC title game. The 1999 season had opened with hope, but star quarterback Vinny Testaverde went down with a torn achilles in the first game of the season. I remember listening to that game on the radio while driving home from going apple picking in upstate New York with my parents. We knew the season was over. But the 2000 season had started well enough, with six wins in the first seven games. They struggled after that, though, and came into this game on December 17th knowing that they had to take care of business and beat the inferior Lions at home. Week 17 would be a road game at the Baltimore Ravens team with arguably the greatest defense of all-time and that would go on to win the Super Bowl that season. They didn't want to leave their playoff hopes down to that game.
Both the Lions and Jets had top ten defenses, and the game became a grind. The Jets led 7-0 at halftime, but the Lions pulled ahead 10-7 on a touchdown with fewer than six minutes to go. The Jets and Lions proceeded to go three-and-out, leaving the Jets to receive the ball via punt with one final chance. They took over at their own 31 yard line with 2:26 to go. They quickly found themselves in a 4th and 1 on their own 40, but needing the first down they gave the ball to future-Hall of Famer Curtis Martin, who pushed two yards for the first down.
Things then opened up for the Jets. A 22 yard pass to Windrell Hayes brought the Jets down to the Lions' 36 yard line with more than 90 seconds to go. Suddenly, the Jets were on the edge of "field goal range", sporting John Hall, who was in his fourth year with the team and who would be a productive NFL kicker for a decade. In fact, the following season Hall would end up hitting an identical 53 yard field goal in the final minute of a game against the Raiders to clinch a Jets playoff berth.
So the Jets, feeling comfortable with their position, suddenly started to get conservative. Testaverde threw a 12 yard completion to Jerald Sowell, but then allowed the play clock to run almost down to zero, allowing the game clock to drop to 40 seconds before taking the next snap. His next pass was a 7 yard screen pass to Curtis Martin, again over the center of the field. By the time they had spiked the ball there were only 22 seconds remaining. Now on 3rd down, Testaverde threw an incompletion, leaving the team with 4th and 3 at the Lions' 17. The Jets lined up for the field goal.... and John Hall missed it.
A sullen Vinny Testaverde took to the podium after the game to talk to the media. "'We were home, we had the crowd on our side, and we had a chance to work up a playoff spirit, but it just didn't happen. We just didn't make the plays." The Jets would lose in Week 17 to the Ravens and miss the playoffs.
What makes that story meaningful? Because it's one of many examples of a billion dollar football team making sub-optimal decisions that lowered their win probability. They were "in field goal range", which means that they were safe and no longer had to take chances to try to score a touchdown. Never mind that a field goal can be missed, or that even making the field goal only gets you to the 50-50 proposition of overtime - seemingly every coach in that situation gets extremely cautious.
So this brings up a question that seems simple, but isn't, and might be an opportunity for a fascinatingly counter-intuitive market inefficiency: Let's say a team is down by three points late in a game and they score a touchdown to go up by three, should they consider missing the extra point to avoid going up by four? Obviously early in the game you want the extra point, but when you're down to a final possession at the end of the game, is the sub-optimal playcalling that a three point lead generates worth giving up a free point?
Well, let's answer that with a chart, using the Pro-Football-Reference Play Index, which includes all data from the past 17 seasons of NFL games. What is the probability of a team winning a game when they start a drive with under two, three or four minutes to go, with deficits varying from between one and seven points (for simplicity, games that ended in ties have been thrown out of this chart):
As expected, your odds of victory decrease significantly as your deficit drops from one to two to three points... but then something funny happens. While you'd slightly prefer being down by three points with under two minutes to go, you'd actually slightly rather be down by four points if you have between two and four minutes to go. In fact, for all three scenarios you'd rather be down by six points than three points.
Let's break down the three and four point deficits a little bit more, by taking the average stats for each drive:
As a further investigation, let's look just at teams down by three points when they start a drive, but compare drives that started with under two minutes to go versus drives that started with under four minutes to go:
So when could teams implement these strategies? Obviously if there are 30 seconds left your opposition is going to be aggressive and desperate no matter what, so you'd rather by up by four than by three. And obviously in the third quarter you'd rather have that point as well. So where are the cut-off points? Well, to do that I broke down the data into 15 second segments. It is plotted below (again, tie games have been thrown out), where each data point represents the 15 seconds up to and including it. So, the data point at "15" means drives that started with 1-15 seconds, the data point at "30" means drives that started with 16-30 seconds, and so on.
Fitting a curve to noisy data is inevitably a bit of a guessing game, and this data is noisy simply because the sample sizes get small at these time intervals, but I fit both curves to third order polynomials, which would allow the curves to have a hitch in them. And that hitch is, indeed, what we see. For teams down by three points, there is a dramatic increase in win probability for receiving the ball up to around 1:30 or 1:45 to go. From there up to nearly five minutes to go, there simply isn't much improvement in win probability.
What the data suggests is that teams are comfortable in a "two minute drill" mindset inside of two minutes. When they have more time than their practiced "two minute drill" they slow down and take their time when they know they only need to play for a field goal.
Perhaps the most interesting bit of data, if you believe the curves, is that win probability actually starts to decrease for drives started when a team is down by four points if there is too much time left. My explanation for that would be that those drives leave plenty of time for the opposition to come back down and score on the other end of the field. They'll only need a field goal to force overtime.
But so while it would be a mistake to over-interpret the fitted lines, the data all are consistent with the idea that being down by three points is a dangerous place to be when you have between two and five minutes left to go. Teams are encouraged, in those situations, to take sub-optimal strategies and to slow down their offense and become cautious.
So would you rather be up by three than up four late in games? Yes, by a little bit. You'd also prefer to be up three than six.
And by the way, the Advanced NFL Stats Win Probability Calculator agrees with me. Assuming two equal teams on a neutral field, a team starting at their own 20 yard line with three minutes to go has a 21% chance to win if down by three, a 26% chance to win if down by four, and a 25% chance to win if down by six. It's not a surprise, since they're working from the same data, but they again agree that you'd rather be up by three points than up by four or six, as counter-intuitive as that seems.
So should teams miss an extra point on purpose when up by three? Yes, though the advantage isn't great. But if you're up by three and in field goal range? You shouldn't take that field goal, even if it's fourth and long. You're much better off trying to keep the clock running by converting on fourth down.
There are a surprising number of these market inefficiency strategies that seem insane when you first utter them and that would cause any television or mainstream newspaper/radio analyst to lose their mind, but which actually make sense when you do the math.
To give a basketball example, Ken Pomeroy argued over a series of blog posts (that all can be found here) that in a tie game with the shot clock off and a reasonable amount of time left, the defense should foul intentionally unless the man with the ball is a well-above-average free throw shooter.
On first glance, Ken's analysis seems nuts. If a guy in a tied game fouls an opponent with 20 seconds left the announcers will fall all over themselves lecturing the player on what a stupid foul that was and how you have to understand the time and situation, when in fact that foul is the team's only realistic chance to win in regulation. If you play that possession out defensively then your best realistic hope is overtime, which is only a 50-50 proposition. You can read all of the details at Ken's site.
There is nothing quite so fascinating in sports these days as the fact that these teams are billion dollar industries, yet they ignore mathematically obvious strategies that would significantly increase their odds of winning. Missing extra points on purpose isn't necessarily so obvious, but not punting on 4th and 1 past midfield in the fourth quarter of a playoff game is obvious, and that didn't keep Jim Caldwell from punting in that situation on Sunday.
It's easy to say that people do stupid things in every industry, and that's certainly true, as we all see every time there's an asset bubble that pops on Wall Street. But sports is unique in the intense media scrutiny on the decision making of coaches and players. When a coach does something outside the box and it doesn't work, the failure is put on the coach. When a coach does something "by the book" and it doesn't work, then it's the fault of the players for "not making plays".
Even Bill Belichick gets roasted in the press (here, here, here, here, here and many more) for making an obviously correct decision, because it happened to not work out. After the game I talked about at the start of this post, nobody criticized Al Groh for his bad decision above to take a conservative approach and leave the game to his kicker. It was his kicker's fault for missing, or so the narrative goes.
Considering how frequently sports managers and coaches are blamed for losses and how often they are fired, what is there possibly to gain from going outside the book? Even if it works out, the media looks at you sideways. And if it doesn't work out, as even sound decisions often don't work out, you are the easy scapegoat.
Ex-coaches love to complain that criticism of their decisions comes from ignorance, and that how dare any of us who haven't walked in their shoes think they've made a mistake. But of course, nobody is saying coaches don't know what they're doing. I'm sure that every coach who has ever held any job on an NFL sideline knows far more about football than me. But they are also under extraordinary pressure to, above all, keep their jobs. And nothing will get you fired more quickly than doing something outside the acceptable box of decisions.
And that is why billion dollar organizations will continue to hobble themselves with sup-optimal decision making.