FanPost

Success and Failure Rates of Top MLB Prospects

Joe Camporeale-USA TODAY Sports

Over the last decade, Major League Baseball organizations have treated top prospects as their most valuable commodity. They are inexpensive (at least for the first six or seven years) and they can produce genuine star players. But of course they can also give rise to failure, lots and lots of failure. Every organization has several prospects that tantalize with scouting reports of potential success, but it seems like they often disappoint. Failure of top prospects is extremely common. At the same time, many top prospects become good, or even great players. So, based on historical precedent, what can we expect from various kinds of highly regarded prospects?

I attempted to answer this question by looking back at top 100 prospects from prior years and see how well they performed in the majors. How often did they succeed? How often did they fail? How often did they become stars? My study attempts to answer these questions by breaking down success and failure rates by rank, position, time period and organization. I think the data provide useful information and shows some interesting patterns in prospect success.

Methodology

For the population of top prospects, I used Baseball America’s top 100 prospect lists from 1990 to 2003. I stopped at 2003 because that is the last data for which the vast majority of prospects have exhausted their cost controlled years. Many prospects showed up on multiple lists, but I counted each occurrence of the player because my goal is to determine the meaning of various rankings by determining their success and failure. Many things can happen to a player ranked #85 in BA’s list. One is that he is never receives a lower rank and then either succeeds or fails. Another is that he can improve his prospect standing and receive a better rank in a later year and then become a success or failure. If I were to only count a player only once, using his best career ranking, then I would be weeding out all of the occurrence of that player at worse rankings, thus artificially taking some very good prospects out of the data sets for the higher rankings.

For each ranking each year, I calculated that player’s average Wins Above Replacement (WAR) from Fangraphs.com over his cost controlled years. If a player totaled fewer than 100 plate appearances or 25 innings pitched in his first major league season, I omitted it from the calculation. If the player also failed to meet those minimums in his second season, I omitted that season as well. I was attempting to account for the fact that many players get very little playing time in their first or second season, and I did not want to give them equal weight in the average WAR calculation. At the same time, I didn’t want to omit all short or partial seasons over a player’s cost controlled years because they are often due to injury or poor performance.

One of the more difficult tasks of analyzing the data was creating an operational definition of "success" and "failure" or what constituted a prospect "bust". Using the rule of thumb breakdown for WAR, I created the following groupings:

Top_prospect_legend_medium

One could argue that I set the bar for major league success either too high or too low. But I think that a prospect needs to become at least an average player in the majors to be considered a success. And I think setting the lower limit for "average" at 1.50 WAR makes sense because it takes into account the common partial seasons early in a player’s major league career that were over the minimum I set above. The data shows that even players in the 1.50-1.75 average WAR group almost always had multiple 2+ WAR seasons, revealing them to have been successful players through most of their cost controlled seasons.

I should make a quick note about language. When I'm talking about "lower ranked prospects," I'm talking about prospects with a rank number which is lower (i.e, #10 is a lower ranked prospect than #20). And "higher ranked prospects" have a rank number which is greater.

Other Relevant Studies

The seminal work on prospect success and value was done by Victor Wang. His initial study, titled "How Much is a Top Prospect Worth?" was published in August, 2007 in The Society for American Baseball Research’s newsletter By the Numbers. Wang looked at the top 25 prospects from Baseball America’s top 100 prospect lists from 1990-1999. He determined success and failure by calculating average Wins Above Replacement Player (WARP). While this statistic is conceptually very similar to WAR, it is calculated using significantly different methodology, and WAR is widely regarded as a superior measure. While WARP has improved in recent years, the 2007 version of the statistic was significantly flawed.

Wang produced a second version of this study in January, 2008 in an article titled "The Bright Side of Losing Santana" for The Hardball Times. This time Wang changed the methodology by using Wins Above Bench (WAB). Again, this is similar to WAR but with a methodology which is both qualitatively and quantitatively different. Also, in this version of the study, Wang looked at all of the top 100 prospects from 1990-1999, breaking them down into four quartiles.

Victor Wang did excellent work in both of these pieces. I believe my work updates and expands on that work by looking at a larger sample of years, using a better measure of player performance, and by breaking down the data to look at player success in several different ways.

Overall Success and Failure Rates

As a jumping off point, I’ll start with the overall success and failure numbers. The data in this table shows the success and failure, to different degrees, of all top 100 rankings.

Overall_top_100_success_rates_medium

My goal with this table was to show what percentage of players failed "Bust", succeeded at all "Success" and became stars (or at least exceptionally good players) "Superior". Fewer than one-third of top 100 prospects succeed in the majors, but there is a big difference between position players and pitchers. While nearly two out of five position players in the top 100 succeed, fewer than one in four pitchers do. And of course many fewer prospects achieve superior success in the majors. A little under one in four position player prospects become stars, while only one in ten pitchers have that level of success.

But of course the overall success rate gives us the most general information. We can get much more granularity by drilling down to see how various rankings and groups of rankings performed.

Prospect Performance by Rank

I’ll start the examination of success by ranking by showing how each of the 100 ranks performed on average. I don’t think the individual data points tell us a lot, as sample size for each rank is small (14) and the variance is obviously high. But I think the trendline is telling, or at least interesting.

Average_war_by_rank_graph2_medium

We can see that average performance drops quickly over the first quintile of rankings and nearly flattens out. But average performance of players of a certain rank does not necessarily equate to success and failure rates. So let’s see how various ranks succeeded and failed.

Prospect Success by Decile

Looking at success and failure rates rank by rank would mean using tiny samples of data from which meaningful conclusions could not be drawn. So I first broke the top 100 prospects down into deciles (equal groups of 10) and determined the success and failure rates of each. This creates a lot of data which we can easily get lost in, but I think the graph makes it easier see what the data represents.

Prospectperfbydecile_medium

The various color bands represent different levels of average performance. The blue through purple bands at the bottom of each column show what percentage of players had various degrees of success. The aqua and beige bands at the top show the proportion of players with various degrees of failure. As one would expect, in general, success rates drop as the prospect ranking number increases. But the decline is anything but steady. It appears clear that prospects in the top 20 have much higher success rates than those of higher rank. And players in the higher half of the top 100 have surprisingly undifferentiated success rates.

While the graph makes visualizing the data easier, I will provide the full tables of the data on busts, successes and superior players, as well as the breakdown by position players and pitchers.

Decile_total_table2_mediumDecile_pos_players_table2_mediumDecile_pitchers_table_medium

That is a lot of data to sift through. But if we focus on the overall success and failure rates, we can see some huge differences between position players and pitchers. Position players ranked in the top 20 had a very high success rate at about 60%, while pitchers in the same group only succeeded about 40% of the time. And position players in the higher ranks level out at a success rate in the mid-to-high 20’s. The success rate for pitchers in the higher ranks levels out in the mid-to-high teens. Because of the smaller samples you get to when you break down deciles into the "superior" group, the variance is high. But we can see in the lower ranks, about 40% of position players become stars, while for pitchers it is more like 20-25%. And while a significant percentage of position players in the higher ranks become stars (as much as 23% in the 70-79 decile), the percentages for pitchers in the higher ranks are consistently under 10%.

These data show a pretty clear pattern of the top 20 prospects performing much better than the rest, with the latter half of prospects performing much worse and at a similar level. But while I think breaking the prospects down into deciles gives us important information, it does lead to some fairly small sample sizes when you further break those groups down by pitchers and position players and look at busts, successes and stars. I think we get a more clear and reliable picture by looking at quintiles, where the patterns become more obvious.

Prospect Success by Quintile

As I did with deciles, I’ll begin with a graph of the data.

Prospect_performance_by_quintile_medium

We can now see clear and meaningful differences between the various rankings. Top 20 prospects have a good deal of success, with more than half succeeding in the majors. Prospects ranked 21-40 have a much lower success rate, under 30%, but have a high percentage of players that contribute in the majors (0.50-1.49 WAR). Interestingly, there is little difference in the success and failure rates of prospects ranked 41 and higher.

Quintile_total_table_mediumQuintile_pos_player_table_medium

Quintile_pitchers_table_medium

As with the breakdown by decile, we see huge differences between position players and pitchers. Top 20 position player prospects succeed at a rate more than 50% higher than their pitching counterparts. The differences between higher ranked position players and pitchers is of a smaller magnitude but it is still significant and it is consistent. Higher ranked position players succeed at around a 30% rate, while similar pitchers succeed only about 20% of the time.

Prospect Success by Position

We’ve already seen that one type of prospects (position players) succeeds at a much higher rate than another (pitchers). But what about other positions? I calculated the average WAR of each ranked prospect by position and these are the results, in descending order of success rate.

Prospect_success_by_position_medium

As expected, pitching prospects are at the bottom of the list. Corner infielders and catchers have the highest success rates with middle infielders having the lowest success rates and outfielders somewhere in the middle. But it is interesting that the positions with the highest success rates do not have the highest proportion of star players. The three position groups that produce the highest percentage of stars are outfield, third base and shortstop. While first basemen and catchers have two of the highest success rates, they have two of the lowest superior rates for position players. Among position players, second base prospects have the least success, but this data is somewhat less reliable because the number of second base prospects in the overall top prospect population is so low (only 42). But this low number reflects how poorly second base prospects are regarded, and I think there is likely good reason for this.

Prospect Success by Time Period

I occasionally hear it argued that we should expect current prospects to succeed at a higher rate because analysts of minor league players do a better job at evaluating prospects now than they used to. So I thought it would be interesting to see if prospect success rates have improved from over the time period this study deals with. So I divided the data into three groups, top 100 prospects from 1990-1993, 1994-1998 and 1999-2003 and compared prospect performance in those three periods.

Prospect_success_by_time_period_medium

Success rates in the latter two time periods appear to be higher than in the first year grouping, but the increase doesn’t appear to be consistent or continual. In fact, the success rate in the third time period is actually lower than in the prior period, but the percentage of superior players is higher. What does this all mean? It appears that the data does not support that prospect analysts are continually improving their evaluative abilities. Perhaps more recent analyses are better than that which was done in the early 90’s, but I don’t see evidence to conclude any farther than that. One could argue that prospect rankings from 2004 to present have been better than those before it, but I don’t think we have enough data to conclude that. I believe 2003 is the last year where we have enough data on the performance of those prospects in the majors to draw meaningful conclusions about prospect success rates. Perhaps in the coming years, as we get more major league data about the major league performance of top 100 prospects from 2004 and beyond, we will see higher prospect success rates, but at present any such assertions are mere speculation.

Prospect Success by Organization

I was curious at how various organizations fared in developing successful prospects, but I think the data is worth little more than entertainment value. First, the sample sizes are all quite small. Most organizations have 20-40 prospects in this study’s population (for the purpose of this calculation, I counted players rather than rankings). Second, I don’t think the numbers tell us anything particularly meaningful. Some of the players were drafted by one organization and developed by another. Some were developed by one organization but played in the majors for another. But entertainment is worth something, so here are the numbers in descending order of success rate.

Prospect_success_by_organization_medium

Conclusion

I think several conclusions are warranted, at least for the period of the study (which includes a great many current major league players).

  • About 70% of Baseball America top 100 prospects fail.
  • Position player prospects succeed much more often than pitching prospects.
  • About 60% of position players ranked in Baseball America’s top 20 succeed in the majors.
  • About 40% of pitchers ranked in the top 20 succeed in the majors.
  • About 30% of position players ranked 21-100 succeed in the majors (with the success rate declining over that ranking range from about 36% to about 25%)
  • About 20% of pitchers ranked 21-100 succeed in the majors (with the success rate declining over that ranking range from about 22% to about 15%)
  • The success rate of prospects (both position player and pitchers) is nearly flat and relatively undifferentiated for players ranked 41-100, and especially those ranked 61-100.
  • Corner infield prospects and catchers are the most likely to succeed in the majors, but outfielders, third basemen and shortstops are the most likely to become stars. Second basemen and pitchers are the least likely prospects to succeed in the majors or to become stars.
  • Prospect success rates have not improved much over time and there is little data to support the contention that prospects are more likely to succeed now than they have in the past.

I do want to make clear that the above numbers are aggregates and therefore they cannot be used to predict the success of individual prospects. For instance if a first base prospect is currently ranked #15, that doesn’t mean that he has a 59.3% of succeeding in the majors. It just means that similarly ranked players have had that kind of success rate in aggregate. Players in that group have ranged from absolute failure to legitimate star status. But I do think the empirical evidence provides a basis for realistic expectations for various types of prospects. No team is going to have all or even most of their top 10 prospects succeed in the majors. Usually, they’d be fortunate to have a third of them succeed. For an historically good minor league system, you’ve got a realistic chance at half of them succeeding in the majors.

Acknowledgements

I want to thank Sky Kalkman and Matt Klaassen for their help in the early conceptual work behind this little project, as well as pointing me towards some existing research on the subject. And thanks to kcemigre for providing me with a big chunk of the data. It saved me a great deal of time and effort.

This FanPost was written by a member of the Royals Review community. It does not necessarily reflect the views of the editors and writers of this site.