Steel Cage Match: Soccer stat showdown of Castrol Index vs WhoScored vs Squawka

Maybe the ultimate exercise in futility is to root for the underdogs in pro wrestling matches. As a kid, I was one of the fools. My favorite wrestler back in the day was Ted Grizzly. He was an older guy who was warm up fodder for Hulk Hogan and the other legends back in the 80s. He wore overalls and a long gray beard and he came looking for blood as if someone had just stolen the gas from his trailer. He never won that I recall, but I remember feeling that sports fan joy when he marched out only to fall just short of victory or get screwed by bad guy foul play. Here's a video of one such abomination, Ted wrestling "Mr. Wonderful" Paul Orndorff. As the commentators astutely point out, Ted should have won by disqualification. Since when are wrestlers allowed to wear casts inside the ring anyway? I guess I still haven't let go.

In some circles its still considered futile to try to describe the beautiful game with numbers. But there is a growing collection of game data (almost all privately held) and analysts who are trying to make sense of the beautiful game through math. One of the popular pursuits is to describe the entirety of a player's performance with one number. We thought it would be fun to see which one is the best. So in honor of those olden golden days and Ted Grizzly, we present the Steel Cage Match of soccer statistics. I know Steel Cage Matches are still common and even more popular with UFC these days, but I'm old school. This is an old school throw down.

There are 3 noteworthy measures that have a goal of defining a player's impact on the game; The Castrol Index, WhoScored Ratings and Squawka Performance Score. While the Castrol Index has been officially producing numbers for MLS since 2011 and FIFA for longer, the challengers published their first MLS numbers just this past season. Here's a little more detail about the numbers.

Introducing the Competitors

In the first corner: The favorite son of FIFA and MLS! The globally accepted, World Cup official champeeeooon score, the Castrol Index!!

And in theeee second corner: Specializing in the in-depth analysis of detailed football data, the WhoooooooScored Ratings!

And in the thiiirrrd cornah!: Squawka, your TV friend that posts live stats while you watch your match! They too have developed a Performance Score for each player and every game!

I'm not going to walk through the methodology of each but you can read their summary in the links. Each of these scores boast using thousands of actions on the field to determine the value of each player. But I will briefly point out a few key differences.

Castrol Index for MLS players ranges from 0 to 810, and both Ronaldo and Messi currently have scores over 1,000. It's an index that is updated monthly and uses a rolling 12 months of data for each player.

WhoScored Ratings is a number from 0 to 10 but most performances are above a 6. Very few players score over an 8. It's a tight range so decimal points matter. They publish scores for each game, unlike the Castrol Index.

Squawka Performance Score adds up a collection of positive and negative actions throughout the game, so the score can be negative. But it also has no upper limit. An average 90 minute player in MLS last year averaged 18 points, to give you a sense of the scale. Their performance score by team is the simple addition of all the player scores. The data is not easily available on their website, requiring a ton of manual work to manipulate. So I am only using their aggregate score in the final cage match.

Tale of the Tape

Okay, let's do one more bit of grounding before we begin. Here are the quintile ranges of MLS players scored in 2013.

A good rule of thumb is that a WhoScored rating greater than 7 or Castrol Index greater than 500 indicates a player performing in the top 20% of the league.

Next let's do an eye test with the Union players and watch them stare each other down. Here are the ratings of last year's Union team. Measure these scores against your personal impressions of the players.

Here are a few of my observations but please leave yours below.

1. Who is the real Jack Mac? It always troubled me that the Castrol Index had Jack in the top 10 in the league. But WhoScored has him as the 140^th best player. Feels to me like he should be between the two scores.

2. The only 2 players with universal love from the stats are Le Toux and Okugo.

3. Casey gets really crushed by Squawka which is a surprise, and WhoScored gives him only an average rating.

4. The Castrol Index says that 7 Union players were in the top 20% in the league. That seems like a lot for an average team. WhoScored only places 2 in the top 20% and that feels more reasonable.

5. For those looking for insight into who should play left back this season, please note that Squawka's score is an aggregate score. If you adjust Fabinho's score for minutes played then he passes Gaddis (and Cruz).

6. I hope Castrol wins. Because otherwise we just traded our best player.

Ok, enough of the eye test; go ahead and pick a winner based on this data before we reveal who we think is best.

The Main Event

Let's keep this simple. A player performance measure should be able to reflect a player's ability to help a team win, no matter their role. Therefore a weighted average of each player's scores by minute should be equivalent to a team's ability to win according to the metric. If it can predict winning then we can assume that the underlying player ratings are therefore more accurate. First let's plot each of the team scores against last season's points.

Couple of observations: Here again DC United confounds the data just as when we looked at possessions (see the team on all 3 graphs with 16 points). Both Castrol and Squawka have the most trouble predicting DC United's horrid results. The Union predictions are much better by WhoScored and Squawka. Castrol over rates the Union substantially by making them the 2^nd best team overall.

And The Winner Is......

Here's a look at the amount of final MLS table variability that is described by the team metric.

WhoScored is the clear winner. That rating reduces the variability left after Castrol's prediction by 33%. Squawka has a lot of work left. Just for fun I removed DC United just to see if the answer would change. And it does. WhoScored is still the clear winner by a similar margin but Squawka's predictability actually passes the Castrol Index. The Castrol Index correlation actually goes down when DC United is removed. Hmm. I think DC United should be kept in the data, but I think that gives Squawka fanatics hope that their rating could improve next year. WhoScored is also a winner because they produce scores for every game, and the data is accessible.

Final Thought

Just two years ago Benjamin Leinwand and Chris Anderson (co-author of The Numbers Game: Why Everything You Know About Soccer Is Wrong) wrote a post that looked at Castrol's ability to predict the final MLS table. At that time the Castrol Index had an Rsquared of about 57%. The authors thought that was pretty good. But as you can see, the Castrol Index has improved dramatically in 2 years, increasing to almost 77%, and will continue to evolve. The arms race for the best performance score will continue on and I'm sure there will be new challengers for the title before too long. But for now WhoScored wears the undisputed Brotherly Game Soccer Stat Belt.

If you've got 13 minutes to kill, here's a video of a classic Steel Cage Match between Hulk Hogan and "the 8^th wonder of the world" Andre the Giant from 1988. Awesome.

Hulk Hogan vs. Andre the Giant 07.31...