NIST IR 7319 - Toward a Standard Benefit-Cost Methodology for Publicly Funded Science and Technology Programs
Executive Summary
The Advanced Technology Program (ATP), part of the National Institute of Standards and Technology (NIST), provides funding on a cost-shared basis for industry-led technology development projects that would most likely not be undertaken by the private sector alone. ATP uses benefit-cost studies as a major evaluation tool for quantifying long-term outcomes and impacts of its more successful projects. These studies generate metrics that directly quantify how well ATP is doing against its mission of accelerating the development of risky technologies that have broad-based economic benefit to the nation.
ATP has conducted and published benefit-cost studies of nearly 30 projects to date. ATP aggregates the measures of net national benefit reported by these studies to present a minimum estimate of portfolio earnings relative to the total costs of the program to date.
Individually, the benefit-cost studies performed to date are consistent with economic models for measuring social and private returns on public and private investment in research and development (R&D). ATP worked directly with the leaders in the field of innovation impact measurement and growth economics to adapt public finance and business models to ATP-funded, industry-led projects. Contractors with expertise in modeling societal economic benefits applied these models in in-depth case studies of ATP projects; the case studies are based on substantial interviews of funded companies, their customers, and industry experts and on other primary data collection activities. All studies are consistent with Office of Management and Budget (OMB) Circular-A94 recommendations for the use of benefit-cost analysis in general; the underlying premise of cash-flow analysis; the use of net present value (NPV) as a key metric of program outcomes; and the Circular's specific requirements concerning features of the analysis such as discount rate, handling of inflation, and sensitivity analysis. ATP staff had considerable expertise in applying benefit-cost analysis in their own research before working at ATP and sought out contractors with similar expertise to perform studies for ATP. NIST economists are leaders in the international standards-setting community. Several ATP economics staff were participants in these NIST activities prior to joining ATP.
Except for studies that cover multiple technology development projects, benefit-cost studies applied to federally funded technology development projects tend to provide a "one-shot-deal" analysis. They are funded at different times in the history of the projects studied and in different years. Not all compute the same sets of metrics. As a result, impact metrics computed in different studies, particularly net present value metrics, are not strictly comparable, and aggregation across studies done at different times presents methodological challenges.
This report seeks to bridge that gap. It culminates a series of efforts to make both existing and future benefit-cost studies of ATP projects more useful to the evaluation process. Previous efforts have included an analysis of studies conducted to date, a workshop of public and private researchers, and a presentation to the American Evaluation Association.
Analysis of the published studies of approximately 30 projects revealed a great many similarities:
- All address ATP's mission. They aim to show, and succeed in showing,
that ATP enables technology development that benefits industry and end
users and generates broad-based benefits to the U.S. economy.
- The studies follow the economic paradigms that emphasize the market
failures in early-stage technology development, the substantial gap
between social returns on technology investment (public plus private)
and private returns alone, and the need to determine instances in which
market failures will prevent this gap from being filled without
government assistance.
- Substantial "bottom-up" interviews are conducted with ATP-funded
company representatives and customers, as well as other market
research, to estimate the benefits of ATP-funded technologies (compared
with existing defender technology as a minimum counterfactual to ATP
funding) and market demand. In general, benefits are estimated on the
basis of a single product unit sold and are extended to an estimated
market. Studies assess the effect of ATP funding on the scope and timing
of the projects.
- Cash-flow analysis techniques are implemented consistent with public
finance literature and good practice in both public and private
investment analysis.
- The studies compute the same basic metrics: net present value, benefit-
to-cost ratio, and internal rate of return.
- Considerable uncertainties exist about the impacts being measured.
Further comparison with the guidelines for benefit-cost analysis provided by OMB Circular A-94 revealed that the published studies were consistent with those requirements.
- The studies emphasize the net present value of outcomes.
- They consistently use the OMB-mandated rate of 7% (real rate) in
discounting benefits and costs to a common base year.
- They consistently estimate benefits and costs in constant dollars,
generally using the dollar value at the time the study was conducted.
- They address uncertainties and include some sensitivity analysis.
OMB Circular A-94, however, does not provide a complete framework for using these studies for evaluating R&D project outcomes and impacts for purposes of retrospective program evaluation of actual program outcomes. It appears to be targeted more to prospective analysis of new federal programs or projects.
The analysis identified four sources of inconsistency in past ATP studies that impede the use of study results, particularly in aggregation.
- Timing of studies relative to ATP funding and the technology life cycle
and the treatment of uncertainties about projected future outcomes.
- Identification of the specific counterfactual to ATP funding and the
related issue of attribution to ATP in cases of multiple funding sources.
- Which metrics to use: Social returns (that is, public and private benefits
on combined public-private investments) and/or public return on ATP
investment?
- Different base years and different constant dollar years: Often base year
and constant dollar year differ in a given study and across studies..
These areas of inconsistency were discussed with workshop practitioners and explored further for the American Evaluation Association's Annual Conference (Toronto, October 2005) and for this report in the context of three broad criteria:
- ATP's evaluation objective of measuring program impacts against the
program's mission while also considering tools appropriate to the
evaluation of public-private science and technology programs more
broadly.
- Quality relative to analytical models established in the economic and
public finance literature and accuracy as provided by established cash
flow analysis procedures used by the business community.
- Effects on consistency and comparability across studies and the ability to
aggregate results of different studies
This analysis yields the following conclusions and recommendations:
- Study timing. Highly prospective studies of project outcomes,
particularly if performed before technical risks and uncertainties have
been overcome and business risks significantly mitigated, may not
generate credible or useful estimates of program impacts even if they
meet high standards of economic modeling and rigor. Probability
distributions of long-term advanced technology project outcomes are
extremely difficult to estimate. Given the uncertainties about these
outcomes, at a minimum some combination of retrospective and
prospective (ex ante) analysis is appropriate as long as the analysis
includes direct evidence of actual commercialized products or processes
that incorporate the program-funded technology.
- Counter factual and attribution.
ATP-funded studies have used various
mechanisms to model incremental benefits of an ATP-funded technology
relative to the counterfactual situation. Mechanisms include acceleration
of benefits and increased likelihood of achieving benefits. Sometimes
achieving any quantifiable economic benefits from a project required
funding from multiple external sources, each of which was indispensable.
A conservative approach is to allocate benefits in some equitable way
among funding sources. Identification and program attribution of
benefits require the following: (1) matching program-funded projects to
direct project outcomes to the extent practical, by tracing product
outcomes backward from existing company products to their R&D
project origins and forward from the ATP-funded projects through the
product development stages to identify the major intersections; and (2)
considering appropriate attribution of all or partial benefits to ATP.
- Which metrics: Social return on investment and/or public return on public (ATP) investment?
ATP has focused on these two sets of
measures. Unfortunately, some studies produced the social return metric,
which compares combined public and private benefits of ATP-funded
technology with combined public and private investments; others
produced just the public return on ATP's investment. (Variants of these
two sets of measures of return appear in the evaluation literature, but
ATP has focused on these two basic themes.) Both sets of measures have
utility, and both have advantages and disadvantages. Combined public-
private benefit-cost analysis is logical for cost-shared technology development projects and helps identify substantial gaps between full societal benefits and private benefits that justify federal funding; however, data collection is difficult. The public return on ATP's investment provides a clearer measure of programmatic impact against its mission of generating broad-based benefit to the nation. The best strategy would appear to be to compute both sets of measures, with attention to keeping the project definitions and traceable benefits consistent for both and to developing practical estimation techniques for computing social return metrics that alleviate data problems.
- Adjusting net present values for timing differences among studies and projects.
Studies performed at different times use different base years and constant dollar years, although ATP's cluster and focused program studies all use a common constant dollar year for a given study and always use the year of first cash flows as the base year of analysis for all projects in the study. Nevertheless, other base year choices might be equally or more meaningful for such studies, for example, the year the portfolio analysis is performed.
In cases in which multiple projects are analyzed in a single study, or in which results of multiple studies are considered together to compute a minimum estimate of portfolio performance, adjustment to constant dollars across the studies will be necessary. For comparability with other studies consideration should also be given to using a common base year for discounting before aggregating across studies. This consideration should include proper interpretation of the discounting effects on metrics that result from different base year choices and an assessment of which approach makes sense for the evaluation objective.
The adjustments are a straightforward process and can be made after studies have been completed using different constant dollar years and different base years for discounting.
ATP's efforts to document and investigate methodological similarities and differences across its benefit-cost studies have demonstrated the utility and commonality of the studies, as well as the complexities beyond pure estimation issues. The challenges do not appear insurmountable. Analysis of key methodological differences among ATP's benefit-cost case studies of nearly 30 projects has helped the program assess the significance of these differences and pose solutions. The goal is to circulate this report to users of benefit-cost results and to benefit-cost analysis practitioners. Feedback is sought and anticipated with the hope that a living, working document will support a more formal, uniform standard-setting process for benefit-cost analysis in the future.
Return to Table of Contents or go to next section.
Date created: July 11, 2006
Last updated:
July 12, 2006
|
