CFB Preseason Rank Analysis

By Michael Gertz
Wednesday, August 29, 2018

Preseason college football rankings have a few issues, and aren't terribly well understood. So just how accurate are they at predicting upcoming seasons?

Issues

Preseason rankings are subjective, which inevitably leaves them open to biases. Worse yet, they also create a psychological anchor which leads to more bias in later rankings. And the anchor is the result of not just a team's own ranking, but that of its opponents. Little updating is done in terms of realizing that the #4 ranked team a team defeated earlier in the season is now #23, and the win wasn't actually as impressive as people made it out to be at the time.

Fundamentally, the preseason subjectivity is acceptable as it is the only way to create accurate rankings before any games have actually been played. But it would probably be preferable if people paid them less attention given the fact that they could influence the rankings that actually end up choosing playoff teams by the end of the season.

The last issue has less to do with the rankings themselves and more about how the public fails to properly interpret rankings in general. It's relatively straightforward that a decent ranking system that ranks a team 50th will have the team ranked too high about half the time and too low the other half. But this breaks down when ranking the top teams, as a "nowhere to go but down" aspect influences accuracy.

Many people assume that a top 2 team has a very good chance of being in the championship game, or a top 4 team has a very good chance of being in the playoff. The recent dominance of Alabama seems to back up that idea, but in reality they have been a rare exception to the general rule that top teams can be expected to finish a bit worse than their preseason ranking.

Accuracy

The problems surrounding preseason rankings are permissible so long as the rankings are relatively accurate. So let's investigate how accurate they actually are. To do this, we've compared each team from the past 4 seasons in the preseason AP Poll top 25 to their final ranking by the Playoff Committee when it comes time for them to select playoff teams.

The main complication is how to categorize teams that began in the rankings but finished unranked. To account for this, we first made a trendline for the odds a team at each preseason rank would finish unranked, as seen in the graph below.

We can see that the very top teams are very unlikely to drop all the way out of the rankings, but by about #8 teams have about a 25% chance of finishing unranked, and by about #17 teams are just as likely to finish unranked as anywhere in the top 25.

We used the #17 break-even point as a guide for quantifying an unranked finish, finding that labeling unranked teams #37 (as opposed to perhaps #26) fit a curve of team expectations where the #17 team on the trendline would finish with a ranking around #25 on average. Below is that graph of what CFP Committee rank can be expected on average based on a team's preseason AP ranking.

Despite the fact that Alabama have started and finished top 4 in all 4 seasons, on average even the #1 preseason team can only be expected to finish about #4. By #5, a #12 finish can be expected. And a preseason #10 on average finishes about #18. Here's how the 2018 preseason AP rankings would look if they were numbered by expected finish rather than current 1 through 25:

4. Alabama
6. Clemson
8. Ohio State
10. Georgia
12. Oklahoma
13. Washington
14. Wisconsin
15. Miami
17. Penn State
18. Auburn
19. Notre Dame
20. Michigan State
21. Stanford
22. Michigan
23. USC
24. TCU
25. Virginia Tech

As we can see, beating any of these teams in the beginning of the season is not nearly as impressive as people make it out to be, and not nearly as impressive as beating an equivalently high ranked team toward the end of the season. It is, however, a good way for a team to get over-hyped early on, which can have a lasting effect going forward.