College Football Playoff: Lessons Learned in Evaluating Teams as a Mock Selection Committee Member

GRAPEVINE, Texas — I was a student journalist at Baylor in 2014 when the final ESPN schedule turned to No. 4 Ohio State. The Bears, infamous Big 12 co-champions with TCU, found themselves out of the first College Football Playoff. Eight years later, much to my chagrin, the College Football Playoff mock draft committee did the exact same thing.

But we’ll get to that.

Twelve members of the media, myself included, were invited to participate in the College Football Playoff’s annual mock draft. The exercise takes place behind the doors of the same posh boardroom at the Gaylord Texan as the actual committee that holds the fate of countless athletic departments in its hands.

For the day, I played the role of Navy Athletic Director and current CFP Committee member, Chet Gladchuk. “You’re acting a little silly to play the role of Chet,” one CFP employee joked to the 72-year-old man whose place I was occupying. But if they wanted it to be kuvaldo, an in-depth look at the controversial 2014 playoff was the perfect place to start.

The challenge to rank the four “best” teams

The process of ranking teams has been reported in detail, but here is a brief description: the rankings are divided into seven groups – every three places 1-9 and every four 10-25. We vote to look at six teams at once for each top three. The first group: Alabama, Ohio, Oregon, Florida, Baylor and TCU. As it should be.

CFP Chairman and NC State Athletic Director Boo Corrigan tasked us with finding the 25 “best” teams in the country. However, it became clear from the beginning that each of us has different definitions of this indicator. Some valued success throughout the season, while others wanted the best team from Selection Sunday. Regardless, the resume proved to be a far more important consideration than which team would win on any given Saturday.

Discussing the criteria with some of the biggest names in the sport was probably the part I was most unprepared for. Former players in the room were adamant that Florida State’s undefeated season was a trump card. I didn’t agree. Try being a fool like me, telling Pro Bowl running back Deuce McAllister and former first-round quarterback EJ Manuel that winning isn’t all about metrics.

The CFP has an analytics system built by SportSource Analytics that can compare up to four teams at once with a wide range of metrics on giant screens in front of the pack. When the Baylor vs. TCU vs. Ohio State vs. Florida State comparison came up on the screen to differentiate the No. 3 and No. 4 spots, everything that happened in 2014 made sense.

Jeyarajah (left) and former NFL linebacker Kirk Morrison

Kevin Jayraj/College Football Playoff

Team schedules are displayed on the page with a list of results and common opponents or head-to-head matches. However, the most impressive part of the page is the color gradient with teams highlighted from green (good) to red (bad). When the schedules for TCU and Baylor came out, the amount of red relative to the rest of the field was impressive.

Never underestimate the power of data visualization!

Additionally, one of the only real criteria listed by the committee is that similar teams with back-to-back wins or conference championships take advantage. CFP Director Bill Hancock confirmed to the group that because Baylor and TCU were featured as Big 12 co-champions, the teams should effectively be treated as having 0.5 titles.

Compared to the outright champions in Ohio and Florida, the decision was not difficult. We can argue about the specific strength of schedule and rating metrics the committee used, but this gave us a complete picture of the committee’s decision.

Possibly led by Manuel’s Florida leadership in the room, the ‘Noles actually jumped to No. 2 in our mock rankings, behind only Alabama. I voted Florida State number 5; The 12 metrics CFP identifies as most related to profit hated FSU and the slim margins of victory weren’t enough to sway me over other (better) conference champions.

Full disclosure, I tended to rely more on performance metrics and game quality than other voters. SportSource Analytics neatly organized and color-coded the stats that correlate most with winning after extensive historical research, with relative scoring and point offense plays leading the way. (A special teams metric was added to the draft committee’s top 12 statistical factors at the behest of former Nebraska coach Tom Osborne.) Other members of the draft committee talked about the quality of their schedule. Some gambled on quarterback play or tried to pitch NFL prospects.

After all, this diversity of opinion is the point. Seven of the 13 CFP committee members are athletic directors. Seven are former college football players, including former NFL player and MIT math PhD John Urschel. Two are coaches. One, former USA Today columnist Kelly Whiteside, is a journalist. Everyone sees the game differently.

The mix of former players, writers and TV personalities gave our group such interesting context. San Diego State legend and NFL veteran Kirk Morrison wanted to make sure the Group of Five contenders got the attention they needed. The Athletic’s Ari Wasserman asked us to weigh in on recruiting gaps. AL.com editor John Talty asked us to remember that Katy Perry was at The Grove for Ole Miss’ game against Alabama. McAllister and former Michigan quarterback Devin Gardner made us think about winning above all else. I mentioned that Ole Miss was blanked 30-0 by Brett Bielema’s Arkansas team, which had a so-called “borderline erotic” bowl victory over Texas. Everyone has their role.

NC State Athletic Director and CFP Chairman Boo Corrigan

Kevin Jayraj/College Football Playoff

How to rank the best from the rest

If you’re wondering why there might be moments of inconsistency in a leaderboard, here’s why. Each secret ballot involves different voters weighing their own criteria while trying to convince other voters that their point of view is correct. And indeed, it made the 7-25 conversation infinitely more interesting and competitive.

“When I was an athletic director at Army, we would be ecstatic to get in the top 25,” Corrigan told the group. “We need to pay attention to the bottom of the rankings as much as the top.”

That’s what we did. The process demanded it.

Some attendees tried to bring the strength of the conference into the picture or reject some programs because they are from the Group of Five. CFP CEO Bill Hancock and Corrigan quickly shot them down. After digging into the matches and seeing the green-to-red gradient on the graph again, it honestly became clear that this wasn’t necessary.

The data visualization had records against teams over .500, against the committee’s previous top 25 and previous top 10, but there were really very few of those high-end data points to choose from. Instead, the graphs that were filled with green caught our attention. Ranked games – or borderline ranked games – were discussed on a case-by-case basis.

We took out Boise State and some were shocked by the amount of quality opponents in a Group of Five conference. Move up. Conversely, Wisconsin’s Power Five program was filled with red. SEC contenders Ole Miss and Georgia were littered with green wins and red losses. Michigan State and Kansas State didn’t have green wins, but their only shortcomings came against the nation’s top teams.

After each round we would vote for new teams. Finally, we had a chance to discuss any major differences. UCLA was originally ranked behind Arizona State, which the Bruins defeated in Tempe, Arizona. This has been fixed. Georgia moved further down, while Arizona moved up a few spots. Marshall took the field after matching up well against teams like Minnesota and Louisville in the 12 factors.

By the way, this discussion lasted five hours. The real thing involves six weeks of talking, many hours of debate and countless weekends of watching clips of every relevant football game imaginable. Hancock noted that there was one committee member who decided to individually evaluate each player in each game he watched. Every year is different.

What will change (and what won’t)

Naturally, that brings us to the 12-team playoff. In many ways, the decisions the commission makes are about to change. With six automatic bids, there will be less focus on determining which teams make the field and more focus on ranking and seeding teams.

When the playoffs expand, the commission will remain at 13 members. They still plan to seed 25 teams and feel comfortable that at least six conference champions will fall into that number. Instead of filling bowl pairings with teams down, they’ll now fill big playoff spots. Basically, the process won’t change much when the playoffs expand, which those in the room hope will be sooner rather than later.

I ended up gaining a lot of respect for the process. It’s not perfect, it’s not flawless, but it’s thorough. With the amount of information and videos available to committee members, I feel more confident that they have all the tools they need to make the best decisions possible. More importantly, the secret ballot process makes it a bit more difficult to determine rankings for branding or matching. Transparency won’t put any conspiracy theories to rest, but it makes me more confident in the process.

And to Hancock, when you need a new name to add to your list of committee members…I’m available.

Leave a Comment