GCR 06-891 - Bridging from Project Case Study to Portfolio Analysis in a Public R&D Program
A Framework for Evaluation and Introduction

Abstract

This paper presents a framework, rooted in the case study method, for evaluating both individual projects and a portfolio of projects. It introduces a prototype evaluation tool that offers new capabilities for the evaluation of a public research and development program in the inter­mediate period before long-run effects can be measured. An assembly of steps produces linked layers of information in a framework that results in a synergistic evaluative capability that enables bridging from project case study to portfolio analysis. This interlinked framework for evaluation can be used both by program administrators, taking a top-down approach, and by project managers, taking a bottom-up approach. It also produces results of interest to policy makers and other program stakeholders. The prototype evaluation tool, the Composite Performance Rating System (CPRS), is designed specifically for the Advanced Technology Program (ATP), but is potentially adaptable to other programs. CPRS uses uniformly collected output and outcome data (indicator metrics) to compute an overall performance rating for each of ATP's completed projects, using a four-star system. The distribution of ratings provides an easy-to-grasp measure of the overall portfolio's performance. This report, organized in two parts, presents in Part I the framework for evaluating individual projects and the entire portfolio, and in Part II a detailed account of CPRS developed specifically for ATP. Part I is expected to be of general interest to the broader evaluation community; Part II, of primary interest to ATP and to other multi-goal, public research and development program administrators interested in new ways of using indicator data for portfolio evaluation.

Key Words: Advanced Technology Program; case study; composite performance rating system; CPRS; indicator metrics; performance metrics; program evaluation; public-private partnership program.

Return to Table of Contents or go to next section.

Date created: June 1, 2006
Last updated: July 3, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)