GCR 06-891 - Bridging from Project Case Study to Portfolio Analysis in a Public R&D Program
A Framework for Evaluation and Introduction

Introduction

Administrators of public programs need evaluation tools that can help them manage their programs effectively and provide answers to stakeholders. A number of evaluation tools exist that can guide management decisions, help assess program performance, and provide answers to a variety of questions. However, available tools are not adequate to meet all the requirements asked of them. Advances in evaluation that increase the understanding of public programs, help determine what is working and what is not, analyze why, and measure effectiveness are important for management, oversight, and public support.

Measuring the impact of the Advanced Technology Program (ATP)² is challenging to evaluators. ATP is a complex program with multiple mission-driven goals. ATP partners with industry to develop enabling, early-stage technologies with the potential for delivering broad-based economic benefits to the United States. As of January 2006 ATP has 768 projects in its portfolio, with multiple pathways of impact—some direct and some indirect—multiple recipients of benefits, and multiple dimensions of success, all played out over a number of years.

Existing evaluation methods are able to answer many questions about ATP's performance and impact reasonably well. Surveys, descriptive statistics, economic case studies, econometrics and sociometrics, bibliometrics, and peer reviews are all evaluation methods and tools applied by ATP to answer stakeholder questions and to manage the program. Evaluation challenges such as measuring spillover effects, capturing portfolio performance in the intermediate period, and other requirements led ATP to also support the extension of existing methods, the development of new and emerging methods, and the compilation of databases to advance evaluative capabilities. The fact that ATP operated under a local climate of experimentation in trying out new technical approaches, and the focus of its parent organization, NIST, on measurement, contributed to a favorable environment for advancing program evaluation methodology, as well as applying it. Now in the middle of its second decade of operation, ATP has strategically deployed a comprehensive evaluation program, and the framework and method presented here are encompassed within that larger evaluation program.³

This report is composed of two parts. Part I presents an eight-step framework, rooted in case study, that substantially extends the evaluative capability of the case study method. The framework provides interlinked, multiple layers of information useful to a variety of program stakeholders, including program administrators, project managers, policy makers, and others. Part I provides an account of how and why the framework was developed within the context of ATP, describes its components, and explains how it can be used to extend understanding of a public research and development (R&D) program. Part I contains Sections 1-3 of the report.

Part II details the development of one component of the eight-step approach to a composite performance rating system (CPRS), which is a key link in bridging from a case study of a project to an analysis of a portfolio of projects. It describes the formulation of CPRS for use by ATP, based on indicator metrics from ATP's first 50 completed projects. It also shows the results of sensitivity analyses, reports the results of applying the CPRS method to ATP, and discusses its strengths and limitations. Part II contains Sections 4 through 9 of the report.

____________________
2. ATP is operated by the National Institute of Standards and Technology, within the Technology Administration of the U.S. Department of Commerce. ATP was authorized by the Omnibus Trade and Competitiveness Act of 1988 (P.L. 100-418), as amended by the American Technology Preeminence Act of 1991 (P.L. 102-245).

3. For an account of ATP's evaluation program over its first decade, its use of existing methods, support of developing new methods, and principal study findings, see Ruegg and Feller (2003).

Return to Table of Contents or go to next section.

Date created: June 1, 2006
Last updated: July 3, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)