Lies and Statistics
A number on this board consider themselves, at least to some extent, sabermetricians. I'm writing this diary in response to what I perceive to be slightly inaccurate interpretations of various sabermetric results.
I will often read blanket statements citing sabermetric results like "clutch hitting doesn't exist" and "managers have no effect." I want to call out the people who say things like this, because these statements in particular are incorrectly stated. In truth, almost every sabermetric result is overstated.
Current research has failed to show that clutch hitting does not exist - it has also failed to show that it exists. We cannot make the claim that it doesn't exist. It is probably less important than once perceived, but the fact is there hasn't been a good study on this.
The statement "managers have no effect" is meaningless without further specification. What people mean when they say this is that in game managerial decisions have little effect. This is mostly due to the fact that all managers basically operate the same way. But, studies on this always take player performance as a given, and ignore the effect managers can have on player performance.
I don't mean to provide the final word on this - only to start a conversation. The point I really want to get across, though, is that it is easy to dilute sabermetric results, take simplified versions of them as axioms, or recast them in ways that aren't correct.
Articles of interest:
Bill James
http://www.sabr.org/cmsfiles/underestimating.pdf
Managers have an effect
http://www.baseballthinkfactory.org/files/primate_studies/discussion/evaluating_managers_part_1_of_2 /
0 recs |
39 comments
Comments
good point
Basically ,yeah, just because you can't quantify something doesn't mean it doesn't exist.
There are things like mental toughness and pressure that do exist. A good situation to envision would be when you are bowling, and you are going for the tough third strike in a row. Sometimes you psych yourself out, and others you concentrate really hard and nail it.
by wildthang on May 23, 2007 8:11 PM EDT reply actions 0 recs
on the other hand
by Moose Tacos on May 23, 2007 8:18 PM EDT up reply actions 0 recs
As far as the numbers go,
I did say that "managers have little effect on the game." Meaning that the amount of games that a manager has a direct effect on is minimal, and at this stage in the game it shouldn't be the highest order concern for a team like the Royals. When we are closer to a pennant and those few games matter, you bet we'll need a manager that can take us to the next level. As long as Buddy doesn't hurt the development of young players, then there is no reason to remove him at this point in the year. Besides they've been playing better, which is a good indication of team development and character. That tone is set by the manager.
by lordbyronk on May 23, 2007 8:26 PM EDT reply actions 0 recs
clutch is dumb
by FireBell on May 23, 2007 8:29 PM EDT reply actions 0 recs
yes, but
by Moose Tacos on May 24, 2007 12:54 AM EDT up reply actions 0 recs
interesting point
i.e., the commentary you'll hear constantly on just about any TV broadcast
re: clutch, sure, its a complex situation and argument, but i nevertheless feel quite comfortable in asserting that the way the issue is usually presented bears little relationship to the actual data
by royalsreview on May 23, 2007 8:32 PM EDT reply actions 0 recs
Right
Nonetheless, to simply argue that there must be clutch hitting or certain managerial moves that are just as important as statistical evidence because there is no good evidence either way is just the classic fallacy of arguing from ignorance. We don't have any falsifiable evidence that Kang and Kodos haven't taken over the leading presidental candidates (insert joke here), but that's hardly an argument we should take that possibility seriously.
Ok, given the current choices, that's not the greatest example, but you know what I mean.
by devil_fingers on May 23, 2007 9:34 PM EDT reply actions 0 recs
Question
And I believe we should fire Buddy Bell, but I'm also not necessarily a sabermetrician.
by Royals Nation on May 23, 2007 10:44 PM EDT reply actions 0 recs
I'll take a stab
Buddy seems to be opposed to any new movement in the game of baseball, in the style of Joe Morgan. In his mind, there is one correct way to play the game, and nothing can change.
I've read quotes where he questioned lifting weights. To me, this is idiotic - nobody can question that strength helps. Similarly, he shows no capacity to adopt any of the truths sabermetricians hold so dear - for instance, that sacrifice bunts early in the game need to end.
In truth, nobody really knows if Buddy is a "good baseball man," or if he sets a good tone for the organization, or if he helps player development. For this we just have to take Dayton's word. But we do know that he is a representative of an ideology sabermetricians cannot tolerate - the unwillingness to learn and adapt.
by Moose Tacos on May 24, 2007 12:48 AM EDT up reply actions 0 recs
Sabermetricians don't say that managers...
And the statistical analysis does support the contention that there is no such thing as clutch hitting. This is an oversimplification of the research, but: How good various players are at hitting in "clutch" situations flunctuates essentially randomly from season to season. That randomness sure does make it look like we're not talking about a real phenomenon.
It's like hitting on Tuesdays. If you ran the numbers over multiple seasons with how guys hit on Tuesdays as opposed to the rest of the week, you'll see some guys doing really well one year and poorly in other years. But the fluctuations from year to year are very random, because players don't reliably hit better or worse on Tuesdays.
by NYRoyal on May 23, 2007 11:48 PM EDT reply actions 0 recs
okay...
Regarding clutch hitting, I highly suggest you read the Bill James article I linked to. Basically, all of the studies you cite were not designed to find clutch hitting - thus they did not find clutch hitting. It is likely that clutch hitting is less important than a lot of people think, but nobody has shown this in a convincing way.
by Moose Tacos on May 24, 2007 12:39 AM EDT up reply actions 0 recs
Slight correction
It is likely that clutch hitting is less important than a lot of people think, but nobody has shown this in a convincing way.
I have read it. Thanks for posting the link. No, the clutch hitting "phenomenon" has not been definitely disproven, but all of the available evidence points against it. Certainly more work should and will be done. But I've yet to see any good evidence to show that it does exist and there is a great deal of evidence showing that it does not exist.
by NYRoyal on May 24, 2007 12:46 AM EDT up reply actions 0 recs
once again ...
Regarding clutchness, this is highly dependent on what you consider "proof" and what you consider "evidence." For someone who believes in the power of statistical methods, it simply does not make any sense to say that clutch hitting does not exist. There is no evidence to support this - all of the available studies have way too much error inherent in their methods.
by Moose Tacos on May 24, 2007 1:58 PM EDT up reply actions 0 recs
And
With regard to clutchness, I'll just say that while perhaps the jury is still out, I have been persuaded by the evidence that I have seen. But more work clearly needs to be done.
by NYRoyal on May 24, 2007 2:38 PM EDT up reply actions 0 recs
Saying that a guy is a 300 hitter
As for clutchiness, have you read Nate Silver's chapter in BPro's book Baseball Between the Numbers? He concludes that "clutch hitting ability exists, more than previous research would indicate. It's about on the order off something like baserunning ability. ..."
by rfloh on May 24, 2007 4:23 PM EDT up reply actions 0 recs
Cool
By no means do I intend to devalue sabermetrics - I am a numbers man through and through, and think rigorous statistics is the best tool we have to understand baseball.
Part of the rigor, for me, is understanding the limitations of the numbers. I think this is so important because I believe that overstatements of the type I mentioned are part of the reason why non-sabermetricians react so negatively to the use of statistics. Also, they are inaccurate, and it doesn't make sense to me to be an inaccurate statistician.
by Moose Tacos on May 24, 2007 12:33 AM EDT reply actions 0 recs
You have to have a balance
by royaldaddy on May 24, 2007 12:03 PM EDT reply actions 0 recs
What is "clutch"
by EricConley on May 24, 2007 1:24 PM EDT reply actions 0 recs
You have to make the assumption
If you set up an artifically defined cutoff, you should see clutchiness occur to some extent no matter what the cutoff is. The effect will be a different size, though.
by Moose Tacos on May 24, 2007 2:06 PM EDT up reply actions 0 recs
the mechanism
As a result of this thread, I've been thinking about the problem of clutch hitting, and I'd like to suggest a different mechanism, which I think might explain the (supposed) phenomenon better: what if instead of clutchness, there was only nervousness. That is, clutch players are not actually changing anything - they are just taking advantage of other players who tense up and don't perform as well. It seems to me that this is a much more persuasive mechanism.
Obviously this doesn't change the fact that it hasn't been shown to exist - the mechanism must be matched by the data.
Another problem with research on clutch: nobody seems to account for the higher quality pitcher usually seen in "clutch" situations. It would be worthwhile to look into this.
Of all situations in baseball, perhaps the most confusing to the sabermetrician is the closer. All outs are the same, in theory, and there is no a priori reason to think you need a special pitcher for close ninth inning games. This led to the idea of "bullpen by committee," an experiment that was aborted after disastrous early results.
It's possible that this idea fell victim to a small sample size - however, it seems that there are some pitchers who adjust better to pitching in "clutch" situations. For the same reason, I think it makes sense that there would be hitters who do the same.
by Moose Tacos on May 24, 2007 2:19 PM EDT reply actions 0 recs
Lots of good points
Another problem with research on clutch: nobody seems to account for the higher quality pitcher usually seen in "clutch" situations. It would be worthwhile to look into this.
It would definitely be worth looking into, but my gut feeling is that on average you might not be seeing higher quality pitchers in clutch situations. Yes, you may be more likely to see the closer pitching in some clutch situations. But, nowadays you almost only see the closer in the 9th inning if the team is ahead by 3- runs. That would define some but not all clutch situations. There are many clutch situations in the 6th, 7th and 8th innings (and perhaps earlier, depending on how "clutch" is defined) where a middle reliever is pitching. And this middle reliever is, perhaps, likely to be worse than the starting pitchers who pitch in earlier innings and non-clutch situations. Just a thought.
Of all situations in baseball, perhaps the most confusing to the sabermetrician is the closer. All outs are the same, in theory, and there is no a priori reason to think you need a special pitcher for close ninth inning games. This led to the idea of "bullpen by committee," an experiment that was aborted after disastrous early results.
This is one of my favorite baseball theory topics. Personally, I don't believe in using a pitcher in the way that closers are typically used. For lack of better terminology, I do believe in the "bullpen by committee". Many people point to the instances when this was used and it didn't work. The problem is that in all of the instances I have seen, they were bullpens with a conglomeration of crappy pitchers. Crappy pitchers are going to give rise to crappy results, regardless of how they are used.
I think saving your best reliever for only 9th inning, 3- runs ahead makes no sense. Even if you allow him to come in and get an out in the 8th every now and then, you're still wasting him. Sometimes the ace reliever should be used as a fireman to come into the game in a crucial situation in the 7th, 8th (or maybe even 6th) inning to snuff out a rally. Also, sometimes a reliever comes into the game in the 8th and he's cruising. If he's dominating, I'd let him pitch the 9th unless the matchups in that inning really don't look good.
In order to test this, even in a non-scientific way, it would have to be tried with a bullpen that has some good pitchers, not just a conglomeration of scrubs.
by NYRoyal on May 24, 2007 2:48 PM EDT up reply actions 0 recs
A middle reliever, generally speaking
by rfloh on May 24, 2007 4:42 PM EDT up reply actions 0 recs
I'm not sure you'll find a stathead
by rfloh on May 24, 2007 4:47 PM EDT up reply actions 0 recs
clarification: stats are awesome
I want to make it clear that I am completely in the sabermetric camp. Probably even more so than NY and NHZ. One effect of this is that I really want people to use numbers correctly - particularly people like NY and NHZ, who have made it clear that they want to use them correctly.
I try to detach myself from the stats at least somewhat during actual games. Especially when I'm playing.
A number of people have said that scouting and statistics must be combined - this is also true, but it is not the argument I am making right now. I am making the argument that sabermetricians must be careful what they say, and have a precise knowledge of what their statistics mean. Otherwise, they dissolve into bland generalities that are no more accurate than what we are arguing against.
by Moose Tacos on May 24, 2007 2:39 PM EDT reply actions 0 recs
I appreciate it
I really do want to understand stats and use them correctly. Please feel free to correct me and educate me wherever possible. [I want to make sure that you know that this was posted without any sarcasm...I genuinely mean it.] I have learned a lot about baseball, stats, sabermetrics and more through forums like this. I know that we all have a lot to teach each other.
by NYRoyal on May 24, 2007 2:54 PM EDT up reply actions 0 recs
Yep
My last SB article is an example of how you can skew stats to further your argument and make no sense. You still have to be smart about how you use sabermetric "tools," sure as sure.
Manangers definitely do have an effect on the game, there's absolutely no question about that. It's just no as big as the MSSM portrays it; second-guessing your average manager (er...Terry Francona?) is a lot less damning than many people think. In the end, you can question bullpen decisions for sure, but it's up to the guys that Francona/Bell/Whoever puts in to actually get people out. It's also hard to measure, from a stats standpoint, exactly what affect "players managers" and team chemistry has on the players' performances.
Clutch hitting exists, no question. If it's 4-2 in the eighth and there's bases loaded, two outs, and Mark Teahen triples in the gap to give the Royals a lead, well of course that's clutch! The thing that I've always argued is that there isn't really an intrinsic "clutch" ability separate from a player's usual abilities. If a guy hits .300/.356/.502 for his career, he'll probably approximate the same rates for his career with runners in scoring position. One season he might hit .239 with RISP, and one he might hit .378, but it usually comes close to evening out in the end.
A huge problem when evaluating "clutch" performances and arguing about the existence of any such tangible ability in clutch situations is "what the heck is 'clutch'," anyway? Is it RISP? I can make a pretty damn good argument that not all RISP at-bats are clutch. It's not a clutch hit to drive in a run to put your team up 17-3, is it?
Is it RISP, two-outs? "Close or late"?
Not only does it make a ton of sense, but it's so much easier to realize this. Take a great hitter, like David Ortiz. Is he clutch? Well, he's had a lot of clutch hits. But his great clutch performances aren't really out of line with his "regular" performance, since his OPSes around 1.000. Is Angel Berroa not clutch? Damn right he's not, but why would he be when he usually OPSes under 700?
Anyway, I'm really just scratching the surface a bit here. Food for thought, certainly; both are interesting topics. I think the trap many sabermetricians fall into is not wanting to deal with either one at all. That's really not much better than believing to your dying day that Kevin Youkilis has no winning intangible.
by NHZ on May 24, 2007 3:24 PM EDT up reply actions 0 recs
ok, ...
the problem with sabermetrics and statistics in general is it is an inductive science. It has no (or at least very little) deductive logic. It takes historical data and draws conclusions from it. This is of course a logic fallacy. Probability theory in general rectifies some of this, but the bottom line is this, all the stats in the word only tell of historical trends, never legitimate predictive models.
How much do you trust what economists say? Sure they have plenty of good ideas to help out at times, but how often do they seam at a loss for words to explain things. They have all these beautiful models that have been created over the last 200+ years but really can be way off from time to time, even in instances that aren't full of unexpected variables. Baseball is probably a more complex system than economics and these sabermaticians have had 20 years to work.
So where do stats come in? Well, as the Bill James article in the OP makes very clear, stability in these numbers are what sabermaticians really look for. the problem with a lot of this is trying to merge the numbers with the real world. Is clutch hitting a stable attribute? well, he says year to year, it is not, therefore he has his doubts on it. but how well can this stat be tried into what is really going through a pitcher and batters head. You can control for every conceivable variable, pitcher quality, PF, and a thousand others with regression analysis, but as I am learning right now, the work to verify accuracy with a single variable is hard, let alone dozens or hundreds.
As I said in my very first post here, I'm a scouting guy. meaningful stats is still a young science in baseball. Before Bill James, how was baseball measured? BA? well, as several of the above posts point out, no one really thinks its the end all stat anymore. Take stats at face value, let all these young companies and organizations who do work in this research this for another decade and see where they are. They tell you what was, not what will be.
Like Spectator said at the start, this site has a lot of "blanket statements citing sabermetric results." Don't ignore, them, just don't think they are the end all of baseball analysis, because what makes baseball, baseball, is its complexities that only a special few ever really master.
non-statistician/economists may stop reading here!
on another note, has anyone else ever done regression work on baseball stats, i have been playing around with some easily available public ones and seeing what I find. If there are any of you, let me know what experience you have cause im think of picking it up as a big time hobby once i graduate. let me know, AIM sn: ZeppelinDZ or email at the same ZeppelinDZ@gmail.com or anyone else who wants to help me procrastinate studying and talk royals.
by ZeppelinDZ on May 24, 2007 5:22 PM EDT reply actions 0 recs
I don't really agree with this.
As for regression analysis...funny you should mention that. I'm a graduate student in anthropology and my thesis work is on baseball. Specifically, I am looking at ethnic and racial identity in baseball from the 60s to today with a special focus on the opportunities and value of minorities (mostly Hispanic...I'll attempt to use other minorities such as African-Americans, but given that race doesn't exist in a biological way...it is difficult to operationally define "Black"). I will be testing things like the marginalization and centrality theories of playing time and player value and as such my thesis will be more sociological and statistical then most anthropologists...which is great for me, since I'm pretty math-minded. Regression analysis will also be used to see general trends in terms of age. I will also be using ANOVA and such tools to test hypothesis of playing time in terms of age and ethnic background.
Anyway, if anyone is interested in this...I was going to post a more complete diary entry about my project as my proposal nears completion. I'd be interested in outside perspective and comments. There are a lot of factors to keep in mind and I'll probably forget something in my first draft.
by dejackso on May 25, 2007 9:45 AM EDT up reply actions 0 recs
the point of inductive logic
As for baseball and economic complexities, I almost took that line out before I posted that. That a discussion too big for an online pub. Maybe some other day, but for now, I'll concede the point.
As for your project, that sounds really interesting. I look forward to reading anything you post on it. I have done a small amount of undergrad research myself on NFL salaries and relative value, but nothing too complex.
by ZeppelinDZ on May 25, 2007 11:40 AM EDT up reply actions 0 recs
Okay...
by NHZ on May 25, 2007 10:26 PM EDT up reply actions 0 recs
Okay, is it a prediction or are you just lucky
by grudz69 on May 26, 2007 12:23 AM EDT up reply actions 0 recs
Hmm...
Science and SABRmetrics do not attempt to prove anything...they attempt to predict the future based on probabilities of the past. The events of the past help shape the future and the processes of the past are often much like the processes of the present. This is how one is able to "prove" theories such as evolution and gravity. We use evidence and statistical models to show the reality of things in the best way possible...we then use new information to update or change our models when necessary.
The truth is that someone who counts cards, or uses statistical models, will "win" more often than someone who doesn't IN THE LONG RUN. So, SABRmetrics is in no way always right...but it tends to be more right than scouting in the long term. Unfortunately, baseball is a sport of the short term...and as such, it's hard to see the power of these statistical models without a long sightedness. In addition, few GMs, owners, and coaches have the time to be long sightedness. Thus, scouting is also important...as it has power as an additional tool.
The other truth is that if we did have all the variables...we'd be able to predict the future with a VERY HIGH percentage. Again, think of something like geoscience. We know how things have happened in the past (uniformitarinism) and so we think we know what will happen in the future. We will probably be right, too. The question of baseball isn't whether we can predict the future with all the variables known...that's 100% true with no real room for debate. The question is whether we can know all the variables...and I think we could probably be safe and say that's 100% impossible. The problem with statistics is that it isn't possible to know all the variables not whether we'd be able to predict the future with all the variables.
by dejackso on May 26, 2007 8:17 AM EDT up reply actions 0 recs
So, if the SABRmetrics predicted that
Since I have a hard time understanding SABRmetrics and can't count cards to save my soul. I will just accept the fate of the pigeon and say a few words in its demise.
by grudz69 on May 26, 2007 2:50 PM EDT up reply actions 0 recs
Long term...
by dejackso on May 26, 2007 8:43 PM EDT up reply actions 0 recs
Ah....
I am not arguing that if you are going to put $50 bucks down on a game that you should look at the numbers. All I am saying is that I look at the game. Our 3rd through 8th hitters went 0-22 tonight. Did the numbers predict that? If so, why would anyone spend money to see the game.
Numbers and their outcome are all well and good, but life and the bounce of the ball don't revolve around numbers. I guess I am rooting for the randomness of breathing. No numbers can predict how many breathes we have left.
by grudz69 on May 27, 2007 12:28 AM EDT up reply actions 0 recs
that should have were... god, proofread....
by grudz69 on May 27, 2007 12:29 AM EDT up reply actions 0 recs
I disagree...
Obviously, the game still has to be played...I wasn't saying that. The point is that one can play the game at max efficiency if one uses the numbers to their full potential. Every event in life has strategies that are better or worse given specific factors. That's the entire basis of evolution after all. Some strategies (biological or behavorial) will be more efficient at certain times and then be less useful at later times. There are people actually working on using baseball as a metaphor for evolutionary behavior theories.
Now many of these strategies may be seen as common sense...and many of them inevitably are common sense. The problem is that many baseball people (Joe Morgan) think that they all are and are scared to use math because they think it will hurt the purity of the game (unlike um... steroids, not having used black players until 1947, the marginalization of hispanic players, uneven playing schedules, etc.) For me, math is beauty. There is something to seeing one's prediction come true...and there is something to seeing one's prediction turn out false. Baseball, unlike any other sport, is math. It is ruled by the numbers more than anything else.
That doesn't mean that all predictions turn out true even when given all the factors. We predict trends and generalizations. No one could have predicted that Randy Johnson would eviserate a bird with a well-placed (or not so well-placed) fastball. People pay to see individual games and statistics predict the larger picture...I think that leaves plenty of room for people to enjoy the "randomness" of the game as well as try to figure out how a team will do in a given season. Remember, that baseball is fluid. While we can predict things given all the factors over a long term, baseball never stays static long enough to have a long term prediction. I might make a prediction of how the Royals will be at the start of the season, but then Berroa finally gets sent down...well, my prediction is off. Again, leaving the randomness of it all.
Oh, as for the life isn't applicable part of my statement. I still hold that it isn't really applicable in the long term...which was my entire point of the last post. In the short term, life is extremely applicable...but birds aren't going to run into Randy Johnson fastballs all that often. Birds will eventually learn to avoid tall people who hate New Yorkers.
by dejackso on May 27, 2007 9:39 AM EDT up reply actions 0 recs
I think we're on the same wavelength
I agree that they don't tell you what will happens and I don't think you will find anyone on this site that would argue that point. However, the information on what has happened is extremely important and the conclusions that are drawn are powerful, useful, and more often than not, reliable.
As for the complexity of baseball, the reason that baseball is simpler is economic is based on one very simple point. Baseball, for the most part, is a series of independent events that have only a limited number of factors that have any measurable effect on the outcome. In this way, baseball can almost be viewed as a lab setting. The social sciences, economics and anthropology in our cases, are not that simple.
by dejackso on May 25, 2007 12:16 PM EDT reply actions 0 recs

by 













