I. INTRODUCTION

Origins

Program evaluation informs program and policy decision making concerning how well a program is working in meeting its goals and objectives. For government programs, these goals and objectives include social and economic outcomes as well as the operations of the program or its immediate outputs. Under procedures to ensure independence, program evaluation may be conducted by independent experts external to the program or by internal program managers. The subset of program evaluation comprising impact evaluation, as compared with ongoing project monitoring and reporting, assesses the net effect of a program by examining outcomes and comparing them with what would have happened in the absence of the program. Benefit-cost analysis directly compares a program's outputs and outcomes with the costs (resources expended) to produce them (Office of Management and Budget [OMB] 2004).

OMB Circular A-94 recommends benefit-cost analysis as the technique to use in a formal economic analysis of government programs or projects (OMB 1992). It states that "social net benefits, and not the benefits and costs to the Federal Government, should be the basis for evaluating government programs or policies that have effects on private citizens or other levels of government." These measurements offer the intuitive simplicity of tools employed routinely in business management and finance. However, the OMB mandate is for public programs to apply a more complex public finance model that extends the analysis to the benefits to society broadly, not just direct recipients of public funds and not to the funding agency itself. Economists have long recognized that there is often a significant divergence between private returns (i.e., the returns to the innovator) and society-wide benefits. The extended public finance models enable systematic quantification of societal returns and the divergence.

Purpose

The Economic Assessment Office (EAO) of the Advanced Technology Program (ATP) seeks to develop a standard methodology for undertaking evaluation benefit-cost studies of science and technology (S&T) projects it has funded for purposes of quantifying the program's outcomes and impacts. ATP seeks to go beyond treating each evaluation case study as a "one-shot deal" and to develop approaches that facilitate comparison and aggregation across different studies. ATP has conducted numerous benefit-cost studies in the past that are consistent individually with OMB Circular A-94 and standard published methodologies. Some studies have analyzed a number of related projects. Although adequate to identify whether individual projects or small groups of projects analyzed together are cost-effective, the net present value (net benefit) impact metrics computed in different studies are not strictly comparable, and aggregation across studies is problematic.

Differences in basic assumptions, timing, and specific metrics used do not adversely affect the value of any individual study, but they hinder the maximum use of the studies by ATP in examining its overall portfolio. In this report, ATP draws on a large body of past benefit-cost studies, the experience and knowledge gained from a workshop with a number of practitioners in June 2004, many hours of discussion with ATP's contractors, in-house analysis, and a presentation to the American Evaluation Association in October 2005 to (1) document consistencies and inconsistencies and sources of incomparability across studies and (2) frame an approach for extending the use of benefit-cost analysis from its typical use in case studies of individual "successful" projects to the evaluation of a portfolio of federal S&T projects.

ATP Background

ATP is a government-industry partnership whose mission is to accelerate commercial development of risky, innovative technologies with potential for broad economic benefit to the nation. Its mechanism is to cost share with private industry the funding of high-risk research and development (R&D) with broad commercial and societal benefits. Through a competitive selection process, ATP chooses projects by applying evaluation criteria:

• Scientific and technological merit (50%)
• Potential for broad-based economic benefits (50%)

Selection emphasizes projects that would otherwise not be undertaken because the risks are too high or because the benefits would not accrue to private investors.
Since its inception in 1990, ATP has announced 768 awards to single companies and joint ventures and has awarded approximately $2.3 billion; industry has provided approximately $2.1 billion as its cost share.

The program anticipates that the major benefits accrue through the creation of new products and processes embodying ATP-funded technologies and their successful commercialization. ATP's EAO seeks to measure outputs, outcomes, and impacts of these technology development projects through a comprehensive portfolio of survey and statistical techniques, case studies, and special issue studies. Generally, case studies, which include high-level overview studies of all completed projects and in-depth benefit-cost studies of selected high-impact projects, are performed by independent contractors.

Methodology

ATP's methodology for this report involves the following:

1. Pursue the program evaluation objective: the evaluation of ATP program impacts against ATP's mission of accelerating the commercial development of risky technologies with the potential for broad economic benefit to the nation.
2. Examine the economics literature for the theoretical rationale and measurement history of social and private rates of return on R&D investment.
3. Consider the OMB mandate for benefit-cost analysis, including specific requirements and general implications for evaluation of public-private S&T projects.
4. Draw on experience and knowledge gained from an ATP workshop that convened private sector and National Institute of Standards and Technology (NIST) practitioners in June 2004.
5. Develop a general methodology from knowledge and experience derived from these external sources.
6. Analyze ATP studies conducted to date for similarities and differences and consistency with this model.
7. Identify key sources of the differences, examine their effects, and consider approaches to addressing inconsistencies.
8. Introduce the initial version of the methodology at a presentation to the R&D Technical Interest Group at the American Evaluation Association in October 2005.
9. Recommend next steps.

In large measure, this report is the text supporting the presentation the author made to the American Evaluation Association at its "Crossing Borders, Crossing Boundaries" Conference on October 26-29, 2005.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)