IV. TIMING AND UNCERTAINTIESS ABOUT FUTURE OUTCOMES

What Are the Issues?

Technology development benefits as measured by outcomes and impacts are highly uncertain and dependent events. At the time that ATP funds a given technology development project, the risks of technical failure (or of meeting only limited technical objectives) are very high. By the end of the period of ATP funding, technical hurdles are reduced; however, even for projects deemed successful technically, generally, significant R&D is still needed before the technology has significant economic value. Market, financial, and business risks remain high even for projects achieving a high level of technical success. Product development phases are often more costly than early-stage R&D financing such as ATP funds. In some technology areas, the time to full commercialization after ATP funding ends can be a decade or more.

Nevertheless, ATP initiated benefit-cost studies early on in the brief 15-year history of the program—partly because the economics staff had significant expertise in these methodologies, but also because of pressure from stakeholders for quantitative program impacts. At the time the earliest studies were conducted, no ATP projects had fully matured. ATP had no experience with the range of outcomes that should be anticipated, and the empirical data available to independent researchers for assessing technology risks and adoption were extremely limited. No one had a good sense of the true probability distributions.

The more recent studies were conducted somewhat later in the technology life cycle. Technology risks had been reduced, and projects studied had made significant progress toward commercialization. All of the studies conducted to date were either entirely or partially prospective analyses. The recent ones have had a retrospective component, and/or barriers to substantial economic impact were considered small. ATP attempts to use results from all of the studies to construct a minimum of the net benefits of the program.

Contrasting Examples

Table 3 illustrates this variability in timing by contrasting two large studies that examined a number of different ATP projects funded in the areas of tissue engineering and component-based software. The two studies were performed by the same independent research organization.

The tissue engineering study was done in the mid-1990s. Performance metrics estimated in the study showed huge impacts; however, the study was conducted very early in the project life cycle of every project covered. Of the seven projects analyzed (all of those ATP had funded in the area of tissue engineering), four had completed their ATP funding period and had significant technical accomplishment; three had just recently started. All faced major continuing technical hurdles, including additional R&D and clinical trials regulated by the Food and Drug Administration. The study included some probabilistic assessment and sensitivity analysis; however, it did not consider the full breadth of possible outcomes or provide a realistic weighting for them. Estimates were based on company interviews and other investigation.

The methodological emphasis was on state-of-the-art modeling of patient benefits and an illustration of the patient-benefit model to ATP's tissue engineering portfolio at the time. The treatment of uncertainty about technical and business outcomes needed for these benefits to be realized was more rudimentary, although it drew on approaches in the economics literature.

The treatment failed to take adequate account of the very early stage, risky nature of these projects. Outright technical or business failure was not envisioned as an outcome. All the projects were anticipated to yield substantial medical benefits. ATP was expected to accelerate the technology life cycle and to improve probabilities of technical success incrementally by increasing the level of R&D funding. Company-provided assessments of progress toward demonstrating technical feasibility (adjusted by progress to date) were used as a proxy for probability of technical success. The schedule for technology adoption adapted an established diffusion model that probably was not adequate for addressing the long-term, high-cost regulatory barriers to adoption of this type of technology.

From hindsight, ATP has learned that typically biotechnologies face a particularly long timeline to commercialization, with ATP funding often coming at an early point in the R&D cycle. A recent survey of the seven projects in the tissue engineering study confirmed that the study was over optimistic. None of the projects had achieved the level of benefits expected by that time. Three technologies now deemed the most successful continue to make technical progress and have entered at least the early stages of commercialization, but benefits are evolving much more slowly than projected for all of them. Three technologies failed to develop; one is in transition to a new company, with its future as yet uncertain.

The component-based software study was performed about five years later. The eight projects analyzed in depth received ATP funding in approximately the same time frame as the tissue engineering projects. Project selection for in-depth study emphasized projects with revenues to date and near-term prospects. All those selected had reached the commercialization stage when the study was performed. Estimated benefits from the eight projects reflected a mix of retrospective and prospective analysis, but with uncertainties greatly reduced for both compared with analyses of early-stage R&D projects.

Theoretically, prospective benefit-cost analysis can model risk through probability distributions and various estimation tools. But pure uncertainty, with outcome distributions largely unknown and in instances in which little empirical data for comparable experience is available, is a different matter. From hindsight, ATP and possibly others have more empirical experience at this point to impose on analyses such as the tissue engineering study; however, the uncertainties are likely so great for studies performed very early in the technology life cycle that results of highly prospective benefit-cost analysis will probably not be very useful for program evaluation purposes. At best, predicted values should most likely cover a very broad range. An accurate probability distribution is likely to generate a low expected value if generated at an early stage.

Modeling and data estimation of prospective project impacts, including uncertainty, can be useful exercises in themselves; however, once such impacts have been estimated, there is temptation to use them for program evaluation purposes without adequate consideration of the original purpose of the exercise or the credibility and/or likely accuracy of the estimates.

As a practical matter, evaluation often cannot wait for fully retrospective results to be captured. Stakeholders need timely results of project and program progress. For R&D projects with a long time horizon, the trail to economic benefits often grows increasingly complex over time—researchers move on, companies disappear, and ATP-funded technology may resurface elsewhere. Program evaluators need to plunge in at some interim point.

Summary

In summary, highly prospective studies performed before technical risks and uncertainties have been overcome and business risks mitigated may not generate performance metrics that provide a credible or useful estimate of program impacts, even if they meet high standards of economic modeling and rigor. Results are likely not comparable to those obtained from retrospective analysis or even prospective analysis performed after technical success has been demonstrated but adoption is still in doubt. However, some combination of prospective and retrospective analysis may be reasonable if uncertainties have been mitigated through time, accomplishment, and substantive interviews with participating companies, their customers, and market experts.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)