 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
|
GCR 06-891 - Bridging from Project Case
Study to Portfolio Analysis
in a Public R&D Program Executive SummaryImproved methods are needed for program evaluation. Program administrators of public research and development programs like ATP, with long lead times to payoff, multiple pathways to impact, and multiple program goals, need more effective tools to manage their portfolios of projects. The challenging evaluation needs of ATP have led it to engage in state-of-the-art application of existing evaluation methods, and to support the development of new and emerging methods as it has developed a comprehensive evaluation program. This report presents in two parts an evaluation methodology developed in the context of ATP's broader evaluation program. Part I presents a framework rooted in the case study approach that makes it possible to bridge from the individual project focus typical of case study to an assessment of a program's entire portfolio of projects. Part II presents in detail an emerging evaluation tool—the Composite Performance Rating System (CPRS)—embedded within the project-to-portfolio approach and critical to accomplishing portfolio analysis. The project-to-portfolio approach of Part I begins with the definition of the portfolio for which the approach will be applied. In the case of ATP, the defined portfolio consists of completed projects for which approximately three to four years have elapsed since ATP funding ended. For projects in the defined portfolio, case studies are conducted, using uniform data collection to build a database of output and outcome data selected to indicate progress toward achieving program goals and referred to alternatively as "indicator metrics." CPRS is constructed to combine selected indicator metrics in order to show overall progress in three dimensions: (1) adding to the nation's scientific and technical knowledge base, (2) disseminating the knowledge, and (3) commercializing new and improved products and processes from the technology developed. The distribution of projects by their CPRS ratings characterizes the overall performance of the portfolio against program goals. The broader framework of which CPRS is a part also provides aggregate statistics on individual indicator metrics and an estimate of minimum net benefits for the portfolio or for the program. Carrying out the project-to-portfolio approach produces interlinked levels of information that can serve as a versatile evaluation resource. Program administrators can take a top-down approach, starting with the portfolio performance distribution and tracing down to the individual project case studies for additional amplification. Project managers, on the other hand, can take a bottom-up approach, starting with the case-study details of projects assigned to them, and tracing up to the portfolio level to find out how their projects or technology areas are performing relative to the broader portfolio. Part II of the report moves from this framework of evaluation to focus in detail on the CPRS tool, which is embedded in the framework. CPRS was developed to allow program administrators to respond to a question of some urgency: How are projects in ATP's portfolio performing overall against ATP's mission-driven multiple goals in the intermediate period after project completion and before long-term benefits are realized? CPRS assigns 0 to 4 stars to each completed project based on overall performance across multiple program objectives, with the weakest performers assigned 0 stars, and the strongest, 4 stars. CPRS allows the portfolio of projects to be characterized in terms of the resulting distribution. For example, application of CPRS to the first 50 completed ATP projects generated the following distribution of projects by performance: 16%, 4 stars; 26%, 3 stars; 34%, 2 stars; and 24%, 1 or 0 stars. This performance distribution was considered in line with expectations of program administrators for the high-risk research projects selected by ATP. Indeed, it is a tenet of the program that some degree of failure must be expected if projects are tackling difficult-to-solve technical challenges. The CPRS star ratings are based on uniformly compiled output and outcome data (indicator metrics), plus an assessment of the outlook for further developments drawn from case studies conducted for all completed ATP projects several years after ATP funding ends. In the intermediate period of focus, evidence that (1) a project added to the nation's scientific and technical knowledge base, (2) the knowledge created is being disseminated to others, and (3) the innovators and their collaborators are moving toward commercializing new and improved products and processes from the project's technology, and the outlook for continuing commercialization efforts is positive is taken as indicative that the project is continuing on the path toward delivering potential economic benefits. CPRS as depicted in Part II has been developed specifically for ATP on an exploratory basis. CPRS represents an initial baseline evaluative tool for assessing overall project and portfolio performance during the intermediate period of an ATP-funded project's life cycle. CPRS would have to be reformulated to be applicable to other public R&D programs with a different mission, goals, and time horizon than ATP, or to rate the performance of projects substantially earlier or later in their life cycles. A search of the literature revealed counterpart development of composite rating systems in a variety of applications. They shared the characteristic of an ad hoc development stemming from the absence of existing theory to direct the selection of variables and the specification of weights used in their construction. Similarly, the formulation of CPRS is necessarily ad hoc in nature—an empirically based, rather than a theoretically based, method. The prototype CPRS formulation was constrained by the kinds of indicator data available for ATP's first 50 completed projects, and influenced by the range of values for those first 50. The use of indicator data means that the CPRS scores signal intensity of progress toward goals but do not reveal estimates of economic benefits. Projects with similar CPRS ratings may differ in their ultimate long-term net economic benefits for this reason and also because project performance may change after the case-study data are collected—for better or worse. Limitations notwithstanding, CPRS provides an easy-to-grasp management and communications tool capable of highlighting overall portfolio performance. The star ratings show which projects exhibit strong and which exhibit weak or moderate outward signs of progress toward program goals during the intermediate period of the project's life cycle. ATP has used CPRS to brief ATP oversight and advisory bodies, public policy analysts, evaluation groups, the broader scientific and technical community, and the general public about ATP 's performance. The CPRS was developed in 2000-2001 using the first 50 ATP projects completed in conjunction with the writing of status reports (mini-case studies) on these projects. Since then, ATP 's Economic Assessment Office has computed CPRS ratings and published over 100 additional status reports. All completed status reports and CPRS ratings can be accessed on a searchable web site (http://statusreports.atp.nist.gov/) and in the following publications:
____________________ Return to Table of Contents or go to next section. Date created: June 1, 2006 |
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov. NIST is an agency of the U.S. Commerce Department |
 |