VIII. CONCLUSIONS AND FUTURE WORK

ATP's efforts during the past two years to document and investigate the significance that methodological differences have across its benefit-cost studies have demonstrated both the utility and commonality of the studies, but also the complexities beyond pure estimation difficulties. The challenges do not appear insurmountable. As for other program evaluation tools, communication and training requirements are significant. However, the basic underpinnings of cash-flow benefit-cost techniques are well established in accounting, finance, and business, and they are adapted to capture societal benefits in public finance and economics literature.

Practitioners have created a body of work that is consistent with good practice as spelled out in the business, public finance, and economics literature. The challenges are in adapting "one-at-a-time case studies" or studies of small groups of projects to a more comprehensive portfolio analysis. The cost and time of doing high-quality studies precludes doing them for all projects funded by a program, never mind doing them in a similar time frame; however, program evaluators and users of study results will need to consider results of different studies at a given time to be able to aggregate and/or compare them. Analyses of key differences across ATP's considerable body of such studies to date have helped us to assess the significance of the differences and to pose solutions.

In general, the programs being evaluated, such as ATP, will not be undertaking such studies themselves but rather will be contracting them to consultants with experience in this type of work. It will be necessary for the program sponsors of this work to communicate the issues to their hired consultants. ATP's experience indicates that the number of consultants with substantial experience in benefit-cost analysis of publicly funded R&D is relatively small; furthermore, even the experienced consultants in this area focus on the study at hand, not how that study will be aggregated with studies done by other researchers or with their own earlier studies. Different experts have somewhat different opinions about solutions to problem areas in which multiple approaches have merit, but inconsistency in approach impedes the interpretation and use of results.

The immediate goal is to circulate this report to users of benefit-cost results and to benefit-cost analysis practitioners in NIST as part of the formal review process and subsequently to ATP contractors and others. Feedback is sought and anticipated with the hope that a living, working document will support a more formal, uniform, standard-setting process for benefit-cost analysis in the future.

Return to Table of Contents or go to next section.

Date created: July 11, 2006
Last updated: July 12, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)