FanShot

Slightly OT - The inside/outside view of forecasting

4

We see this all the time when evaluating players and teams, but here's a Nobel Laureate discussing it in a non-baseball setting: After meeting every Friday afternoon for about a year, we had constructed a detailed outline of the syllabus, written a couple of chapters, and run a few sample lessons. We all felt we had made good progress. Then, as we were discussing procedures for estimating uncertain quantities, an exercise occurred to me. I asked everyone to write down their estimate of how long it would take us to submit a finished draft of the textbook to the Ministry of Education. I was following a procedure that we already planned to incorporate into our curriculum: the proper way to elicit information from a group is not by starting with a public discussion, but by confidentially collecting each person’s judgment. I collected the estimates and jotted the results on the blackboard. They were narrowly centered around two years: the low end was one and a half, the high end two and a half years. .... Then I turned to Seymour, our curriculum expert, and asked whether he could think of other teams similar to ours that had developed a curriculum from scratch. Seymour said he could think of quite a few, and it turned out that he was familiar with the details of several. I asked him to think of these teams when they were at the same point in the process as we were. How much longer did it take them to finish their textbook projects? "You know, I never realized this before, but in fact not all the teams at a stage comparable to ours ever did even complete their task. A substantial fraction of the teams ended up failing to finish the job." This was worrisome; we had never considered the possibility that we might fail. My anxiety rising, I asked how large he estimated that fraction was. "About 40 percent," he said. "Those who finished, how long did it take them?" "I cannot think of any group that finished in less than seven years," Seymour said, "nor any that took more than ten." .... This embarrassing episode remains one of the most instructive experiences of my professional life. I had stumbled onto a distinction between two profoundly different approaches to forecasting.... The inside view is the one that all of us, including Seymour, spontaneously adopted to assess the future of our project. We focused on our specific circumstances and searched for evidence in our own experiences. We had a sketchy plan: we knew how many chapters we were going to write, and we had an idea of how long it had taken us to write the two that we had already done. The more cautious among us probably added a few months as a margin of error. But extrapolating was a mistake. We were forecasting based on the information in front of us, but the chapters we wrote first were easier than others and our commitment to the project was probably at its peak....The argument for the outside view should be made on general grounds: if the reference class is properly chosen, the outside view will give an indication of where the ballpark is. It may suggest, as it did in our case, that the inside-view forecasts are not even close.