NIST GCR 05-879 —Photonics Technologies:Applications in Petroleum Refining, Building Controls, Emergency Medicine, and Industrial Materials Analysis

2. ANALYTICAL FRAMEWORK AND METHODOLOGY

ANALYTICAL FRAMEWORK

This study uses microeconomic, cash flow-based case study methodology to generate an objective assessment of broad-based economic benefits relative to investment costs. The framework includes both in-depth case study evaluation of individual ATP-funded projects and a broader, higher level, multi-project portfolio, or cluster approach. This hybrid approach yields quantitative measures of project and portfolio performance familiar to the financial and economic communities—net present value, benefit-to-cost ratio, and internal rate of return—as well as qualitative descriptions of other non-quantified benefits.

Cluster Approach: Case Studies and Overview Studies

The cluster approach used in this study is a hybrid of two traditional approaches for evaluating the benefits of publicly funded science and technology programs: detailed case studies of individual projects and overview studies of many projects.

• Detailed case studies are used to generate rich insights into complex causal chains from innovation to market pathways to public benefits. Based on an understanding of causal chains and industry dynamics, case studies can be used to develop cash flow estimates of the value of technology innovations.
• Overview studies of many projects identify observable trends in the flow of public benefits from technology innovations, without developing detailed information about specific technologies, industry, and market factors. While overview studies can provide useful general information about the benefits of technology innovation, they do not generally lend themselves to developing quantitative benefit time series.

The hybrid approach helps to realize the advantages of both detailed case studies and multi-project overview studies while avoiding their limitations.

Public and Private Benefits

For purposes of evaluating the performance of publicly funded science and technology projects, the benefits of technology innovation can be segmented with reference to different classes of beneficiaries.

• Private benefits: Economic benefits enjoyed by the innovating firms funded on a cost-share basis with the ATP (or another public agency) are considered private benefits. The innovating firm's expectation that these private benefits (profit contribution from new technologies) will be realized is a necessary precondition for completing remaining technical development tasks (after the successful completion of the ATP-funded project phase) and for undertaking subsequent commercialization to facilitate economic and social benefits from ATP-funded technology projects.
• Public benefits: Economic and social benefits arising from the ATP-funded technologies and enjoyed by downstream industrial firms and end users of industrial products are considered public benefits. In microeconomic terms, public benefits represent spillover phenomena, where the degree of spillover represents that portion of total benefits that the innovating firm is unable to capture for itself (Jaffe, 1996). As suggested by both the theoretical and empirical economics literature, public benefits, or the extent of spillover from ATP's investment, can be expected
  to substantially exceed the magnitude of private benefits (Mansfield et al., 1977).

In addition to economic benefits for industrial users (eventually allocated among producers and consumers based on market forces), many important public benefits have a public goods quality that society will enjoy directly. Examples include:

• Reduced harmful environmental emissions
• Improved clinical outcomes for medical patients
• Conservation of scarce energy resources
• Knowledge diffusion about new technologies

The flow of public and private benefits to different classes of beneficiaries is illustrated in Figure 1. This model is sufficiently general that it could be applied to many federal science and technology evaluation efforts across different agencies operating under a range of science and technology missions.

Figure 1: Flow of Public and Private Benefits from ATP-Funded Technologies

Retrospective versus Prospective Benefits

The temporal placement of benefits is an additional important variable for science and technology benefit analysis, and the cluster study utilizes retrospective as well as prospective analysis, as appropriate.

• Retrospective benefits: Realized benefits (to date), identified on the basis of well-documented analysis, involve less uncertainty and can lead to higher levels of confidence in the analytical results of science and technology impact studies than prospective future benefit estimates.
• Prospective benefits: Expected future benefits are typically associated with a variety of risks and uncertainties, including market introduction, consumer acceptance, and manufacturing ramp-up. While these risks and uncertainties may be mitigated through various means, including documented expressions of interest from potential customers, prototype sales, rigorous market studies, as well as reasonable expectations of near-term commercialization, the inherent uncertainties of prospective analysis necessitate the use of expected value constructs, where first-order benefit estimates are scaled down with consensus-based probability estimates.

Benefits Resulting from ATP Investments

The central objective of this cluster study is to gauge the programmatic impact of rigorously screened and well-timed ATP investments. This requires a fair assessment of the portion of public returns attributable to the ATP.

To this end, benefit analysis includes a consideration of the contributions of other sources of public and private funding to ultimate benefits. As an example, ATP-funded high-risk technology projects may be preceded by basic and applied research projects funded through the NSF or other U.S. agencies. Alternatively, agency funding for subsequent technology and product development may be put in place after an ATP cost-shared project is completed. In either case, a series of sequential investments by the ATP and other agencies will tend to enhance the value of prior investments relative to the technology's potential to generate economic and social benefits.

The appropriate attribution to ATP will depend on specific circumstances of each case, reflecting the relative importance as well as the size and timing of ATP and other investments. In general, benefits attributable to ATP represent the increment over the counterfactual situation where ATP funds were not made available to support the specific project.

BENEFIT-COST ANALYSIS METHODOLOGY

Two projects were selected for in-depth case studies to document project history, including an account of the need for ATP funding, the characterization of technical challenges and accomplishments, opportunities for commercial applications, pathways to markets, and an identification of all expected benefits, both public and private.

For those benefits that can be meaningfully quantified, cash flow estimates are generated for a conservative base-case scenario and for a more optimistic step-out scenario, incorporating higher unit sales projections and higher per unit benefit estimates, as appropriate. Economic losses to defender technologies being displaced by new ATP-funded photonics technologies are estimated and subtracted from cash flow benefit estimates to arrive at net benefits.

The focus in this report is to gauge ATP's impact and effectiveness through performance metrics that capture the value of public benefits attributable to ATP's investment. Benefit cash flow estimates are compared to investment costs to compute public benefit performance metrics, including benefit-to-cost ratios, net present values, and internal rates of return.

Benefit cash flow analysis can also be expanded to include private benefits to the innovating firm as well as all public and private investments to yield a more inclusive and broader measure of social rate of return, as described in the box below.

Mini-studies for three additional ATP-funded projects were completed and resulted in the identification of qualitative benefits.

Performance metrics for public returns were computed on the basis of benefit cash flow estimates from the two case study projects against public investments in the five projects. Metrics are conservatively estimated and represent expected lower bounds.

Future economic studies may document and estimate benefit cash flows from the other three projects in the cluster and result in substantial upward adjustments of performance metrics.

In addition, a broader social rate of return was computed for the two individual case study projects.

Return to Table of Contents or go to next section of report.

Date created: July 12, 2006
Last updated: September 14, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)