4. ERRORS IN FORECASTING DEVELOPMENT COST, TIME, AND PROFITS

Forecasts of the private rate of return from an investment in new technology depend on, among other things, how much will be spent on development and how long. Development will take, as well as on the probability of technical success, commercialization, and economic success. Further, it depends on the profitability of the new products and processes, if any that stem from the investment.

Probably the most important reason why sophisticated models have proved so difficult to apply is that firms have not been able to make reasonably reliable forecasts of-development cost and time, or the probability of success, or the profitability of new products or processes. During the 1960s and 1970s, my students and I collected a substantial amount of detailed data regarding the size of errors in forecasting these items. Since these data seem to be all that are currently available, I have no choice but to use them, although the need for updating them is clear. However, for what it is worth, a small sample of R&D executives (that I interviewed in connection with this paper) felt that the situation in this regard had not changed very much.⁸ Indeed, many of them thought that such forecasts tend to be poorer now than in the 1960s and l970s, because down-sizing at many R&D laboratories has lowered the capability to do such forecasting.

Development Cost and Time

To illustrate our findings regarding development cost and time, consider the results of an early study in the drug industry (Mansfield et al, 1971). Very detailed data were obtained for a major ethical drug firm and a major proprietary drug firm concerning the errors in the cost and time estimates made at the beginning of drug-development projects. For over 80 percent of the projects in the ethical drug firm, the actual cost and time exceeded the estimated values. The average ratio of actual to estimated cost was 1.78; the average ratio of actual to estimated time was 1.61. Cost and time estimates were less reliable for new chemical entities than for compounded products and alternate dosage forms. In the proprietary drug firm, the average ratio of actual to estimated cost was 2.11, and the average ratio of actual to estimated time was 2.95. Again, the overruns were greater for more ambitious projects.

When we compared the overruns in these two drug firms with those in weapons development, we found that the cost overruns for new drug products were less than those in weapons development and that the time overruns were greater than in weapons development. However, it is important to note that the cost overruns in the drug firms began to approximate those for military projects when entirely new types of projects or larger technical advances were attempted. For example, the average ratio of actual to expected cost was 2.25 for new chemical entities in the ethical drug firms, 2.75 for new products in the proprietary drug firm, and 3.2 for a sample of airplane and missile projects. Turning to time overruns, we found the average ratio of actual to expected time was 1.89 for new chemical entities, 3.24 for new products in the proprietary drug firms, and 1.4 for the airplanes and missiles.

In the ethical drug firm, we tested various hypotheses concerning the effects of various factors on the size of a project's cost overrun. In accord with these hypotheses, it turned out that technically more ambitious projects tend to have greater cost overruns than technically less ambitious projects. Also, products with wider spectra of activity tended to have larger cost overruns than single-market products, and projects with small estimated costs or longer duration tended to have larger cost overruns than projects with large estimated costs or shorter duration. In the proprietary drug firm, there was also a significant tendency for technically more ambitious projects and projects with smaller estimated costs and longer duration to have larger cost overruns.

When the same kind of model was used to analyze development time, the results were rather similar to those for development cost In the ethical drug firm, there was a significant tendency for products with wider spectra of activity and projects with smaller estimated lengths to have greater time overruns. In the proprietary drug firm, there was also a significant tendency for projects with small estimated lengths to have greater time overruns; moreover, there was a nearly-significant tendency for new products to have larger time overruns than product improvements.

Probability of Technical Completion

Let's turn now to estimates of the probability of technical completion of R&D projects. To illustrate our findings, consider the proprietary drug laboratory cited in the previous section, where records of many completed projects included an estimate of the probability of achieving the technical objectives, as stated in the project proposal. This estimate was made at the time of formal project proposal. Data for 79 completed projects indicated that the estimated probability of technical completion was on the average , a very good indicator of actual outcome, the average estimated probability of technical completion (0.81) being very close to the actual proportion of projects that were completed (0.76).

However, the fact that the average estimated probability of technical completion was close to the actual proportion of projects that were completed does not mean that the estimates were useful in predicting which projects were more likely to be completed. In fact, the estimated probabilities of technical completion were of some use in predicting which projects would be completed and which ones would not. But they were not of much use. Even if they were employed in such a way that the probability of an incorrect prediction was minimized, they predicted incorrectly in about 30 percent of the cases. (One would have expected to have made incorrect predictions in only 36 percent of the cases by chance.) We also compared the actual proportion of projects that were technically completed with the average estimated probability of technical completion for projects attempting small, medium, and large technical advances. For those attempting small technical advances, the estimated probability of completion, on the average, overstated the risk of failure. On the other hand, for those attempting medium or large technical advances, the estimated probability of completion, on the average, understated the risk of failure.

Discounted Profits from New Processes and Products

To complete this brief survey of our findings during the 1960s and 1970s regarding forecasting errors, let's turn to the accuracy of forecasts of discounted profits from new processes and products. Consider one of the largest firms in the country.⁹ During the 1960s and 1970s, this firm made a careful inventory of the major new products and processes it developed each year and forecasted the discounted profits from each such technological development. Moreover, in each subsequent year, it revised these forecasts in the light of new information. For example, after making its initial estimate in year 2 of the discounted profits from each process or product developed in year 1, revised estimates of this sort were made in year 3, year 4, year 5, and so on. Because these estimates were systematically and carefully updated, they provided a relatively unique opportunity to study how quickly forecasts of this sort converge on their true value. Unfortunately, because of reductions in available resources, this analytical effort was reduced by the firm during the 1980s, and terminated in the early 1990s.

Based on an early study of this firm's experience (Beardsley and Mansfield, 1978), there frequently have been rather significant revisions of the profit forecasts during the first five years after a new product or process has been developed, but, as one would expect, these revisions have become more minor as time goes on (and more and more of the uncertainties have been resolved). By nine years after the development process ends, it appears, in this firm at least, that a reasonably definitive estimate can be made of the discounted profits from the new technology. In very few cases were any significant revisions made in this profit estimate in the tenth to thirteenth year after termination of development. Thus, we could safely use the estimate of a new product's or new process's discounted profits made nine years after its development as an adequate approximation to its actual discounted profits.

To see how rapidly the forecasting errors diminish as time goes on (after the development of a new product or process), we divided the forecast made one year -later, two years later, and so on, for each product or process by its actual discounted profits (i.e., the forecast nine years after development). Then, as a simple measure of the size of the forecasting errors, we calculated the proportion of cases where this ratio was greater than or equal to 2.0 or less than or equal to 0.5. Figure 1 shows the decrease that occurred in this measure of forecasting error as one revision after another was made in the forecasts. The initial forecasts—those made one year after development—were generally quite poor, the proportion in Figure 1 being 0.50 for processes and 0.62 for products. During the first four years after the initial forecast, the size of the forecasting error, as measured by this proportion, decreased at a relatively constant rate. By five years after development, this proportion was 0.06 for processes and 0.15 for products.

Figure 1 shows that the initial profitability estimates for individual product and process innovations developed by this firm were not very accurate. To see whether the firm's initial estimates improved over time, we categorized the 57 new processes and products by the year they were developed, and calculated the frequency distribution of the ratio of initially forecasted to actual discounted profits in each such category. The results indicated that there was no tendency (at least during this period) for the initial estimates to improve over time.

Figure 1 - Proportion of new processes or new products in which the ratio of forecasted discounted profits to actual discounted profits was greater than (or equal to) 2.0 or less than (or equal to) 0.5, 57 new products and processes, 1-9 years after product or process development.

To see how closely correlated the initial forecasts were with the actual discounted profits from each of the new processes or new products and to estimate the relationship between them, we regressed the initial forecasts on the actual discounted profits. For new products there was a surprisingly low correlation between the initial forecasts and the actual discounted profits, R² being only .14. In the case of new processes, the correlation was much higher, R² being about .87. Thus, the initial forecasts for processes tended to relate more closely to the actual outcomes than those for products. Both for processes and products, the initial forecasts tended to be relatively optimistic in cases where actual profits were small and relatively pessimistic in cases where actual profits were large. Specifically, the forecasts underestimated the profitability of new processes that had discounted profits exceeding about $3 million and of new products where they exceeded about $1 million, and over-estimated the profitability of less profitable new products and processes.

Return to Table of Contents or go to next section.

Date created: June 15, 2006
Last updated: June 16, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)