On predictions, evaluations, and what we don't know

At my alma mater of William Jewell College, there are three questions that powered its liberal arts core curriculum. The first question of 'what is real?' is a philosophical black hole of a pondering that can result in every conceivable answer under the sun, if the sun is indeed real, for which there may or may not be any evidence, and if words convey actual meaning if meaning is indeed a thing or if words are just an arbitrary set of sounds that SYSTEM FAILURE SYSTEM FAILURE

The second question is noticeably simpler--'how should we live?' It's a difficult question, don't get me wrong; but its difficulty lies in the range of possible answers, not whether or not 'living' is a thing or not. Clearly, part of the answer is to be a Kansas City Royals fan and not a St. Louis Cardinals fan.

The final question--'how can we know?'--has fantastic applications to any line of work, thought, or creed, and can be asked at any step of the line from prediction to evaluation. It is a challenge to be responsible in the decision-making process, to draw conclusions from the best available data.

In baseball, 'how can we know?' has been one of the core questions of its existence. Baseball is filled with stats, and the sabermetric movement arose because curious individuals chose to logically follow that query as far as it extended. Better evaluations lead to better predictions, and vice versa. Statisticians are still following that question today, as new stats and technology let us dig even deeper into the endless mountain that is baseball. How can we know?

The Kansas City Royals have just won the 2015 World Series, conquering three excellent teams on their road to glory. In hindsight and on a surface level, the returning American League Champions making the final step to win the Series is not surprising in the least. But hindsight is not a luxury afforded before the event in question.

Everyone who enjoys sports likes to predict stuff about the sports that they enjoy. This is why NCAA Tournament brackets are so fun, it's behind why fantasy sports are so big, and it's how Mel Kiper can rock a haircut from a bad Star Trek episode year after year and get paid for it. Here at Royals Review, we also predict things. We did so on March 30, with the subheading of 'These will probably all be wrong, so bookmark this to yell at us in October.'

That sentence could not have been more correct. Our predictions were an unequivocal failure.

The Royals went 95-67, winning the division. Lorenzo Cain was their best position player (6.9 average WAR between Fangraphs and Baseball Reference), and their best pitcher was Wade Davis (2.7 average WAR). Our predictions:

Only one of us (myself) predicted within 10 wins of the correct win total, only one of us (Connor) predicted the player of the year correctly, and only one of us (Duggan) predicted the pitcher of the year correctly.

If you think zooming further out would give us more clairvoyance, you would be as mistaken as if you thought you could find parking in downtown at noon during the parade. The AL division winners were the Royals, Toronto Blue Jays, and Texas Rangers; the wild cards were the New York Yankees. The NL division winners were the New York Mets, St. Louis Cardinals, and Los Angeles Dodgers; the wild cards were the Chicago Cubs and the Pittsburgh Pirates. Our predictions:

The eight of us predicted each playoff spot for both the AL and NL, totaling 80 predictions. We got 19 correct, or basically the success rate that 2013 Mike Moustakas had in hitting the ball. 2013 Mike Moustakas should not be a thing for which to strive. It should be a shameful thing, like that one time when you threw up at the family Christmas from drinking too much egg nog. Our non-NL Central/NL West success rate was 6 out of 64, or basically what would happen if 2013 Mike Moustakas batted after drinking too much egg nog.

Now, we could just be really terrible at predicting things. Our 2014 predictions were actually pretty good, though, as far as these things go; five of eight predictors pegged the Royals' win total within five games. Still, that could be just luck and we are really terrible at predicting things.

But ESPN analysts should be really good at predicting things. After all, ESPN is the World-Wide Leader in Sports, and they are so confident in their ability to outpace others in viewership that they willingly employed Harold Reynolds as a cruel joke for a decade.

So when 15 ESPN experts get together to predict the MLB season, you know some truth is being seared onto the digital page.

If these predictions look to you more like a flaming pile of dinosaur feces than the predictions of people who are paid six figures to predict, analyze, and discuss sports things, you would be correct. Out of 150 possible correct answers, these experts got 36 correct. That's good for a 24% success rate, or just as good as what eight dudes with a computer, brains, eyes, and some pop-tarts can do on the side for a baseball website that is obsessed with otters.

These 23 individuals were noticeably low on our Kansas City Royals in a consistent manner, one in which the projection systems agreed. Before the season, the PECOTA projections computed that the Royals win 72 games, a good 23 games short of their regular season win total. Fangraphs predicted for the Royals to win more games...but only seven more, at 79 games.

So what's with all this terrible predicting? How can we know? Can we ever know?

There will always be unexpected variance. Maybe it's a player like Mike Moustakas making a hitting adjustment and reaping huge and instant benefits, or maybe it's an entire team that passes injury around like the flu. The sheer amount of human input required for a sport like baseball will always yield randomness at some level.

There is another item at play, and it probably doesn't get enough credit: there is so much we don't know. Sabermetrics has made huge leaps and bounds since ye olde days of Bill James and moon landings, but it's still learning about certain things like pitch framing and shifts. Pure statistical projection systems were so laughably bad in predicting the Royals' 2015 that it can be difficult that they are usually very good.

But we don't even have all the data that is available. Every team has their own analytics department, and some of them have proprietary software, algorithms, and methods of prediction. We know that Statcast data exists, but it is not publicly available in full. Even beyond statistics, we are not privy to the ins and the outs of free agency, trade deals, and the workings of each front office. A deal that makes sense to us may not b realistic for a myriad of reasons, but we will never know that. Decisions made on the field, such as Mike Jirschele's sending of Lorenzo Cain home on Jose Bautista's throw to the infield in the ALCS, only happen with advance scouting that we do not have. Medical results on players are also private.

Does this mean we can't criticize moves made by a General Manager or coach? Absolutely not. But the success of the Royals this year, along with some terrible predictions made by both experts and amateurs, suggests that front offices have more information than we do. What we do not know may be known by others, and perhaps those others are the people in charge of those decisions.

How can we know, though? The short answer is: we can't. Still, we shouldn't throw good analysis to the wind. But we ought to consider that we aren't quite as smart as we think.