Part I
A Framework Linking Evaluation of Individual Projects to Assessment of a Portfolio of Projects

Traditionally, project case study and program portfolio analysis have been pursued as separate lines of evaluative inquiry. Project case studies are commissioned by nearly all public R&D programs. They offer rich, detailed information about a project or cluster of projects, but limited information about overall program performance. In contrast, evaluative methods aimed at portfolio analysis, such as survey, focus on a whole body of projects, but, by necessity, eliminate individual project details in favor of overall measures of a program's performance.

Program administrators, policy makers, and other stakeholders typically need information at both the project and the portfolio levels to understand a program, manage it, and report on it. They need both the details and the overview. Ideally, they would be able to traverse smoothly between the project, or bottom level, and the portfolio, or top level of their programs. But, heretofore, a tight linkage between the bottom and top levels has not been provided in the evaluation of public R&D programs.

For public R&D programs with new project starts each year, multiple program goals, multiple years of funding per project, multiple beneficiaries from project success, and multiple paths of impact, years must elapse before it is possible to measure retrospectively long-term impacts for a portfolio of projects. To help inform program developments during the inter­mediate period—after government funding of research has ended, but before long-term impacts have been realized— ATP adopted several evaluative strategies during the 1990s. One strategy was to conduct prospective case studies, which attempted to broaden the analysis from single projects to clusters of projects.⁴ Another strategy was to conduct surveys that included questions about accomplishments, plans, and expectations. A related strategy was to aggregate selected individual output and outcome measures that relate to program goals and use them as "indicator metrics" to signal project and program progress.⁵

Partly through forethought, and partly through a series of evolutionary steps, ATP developed a framework linking evaluation of individual projects to assessing a portfolio of projects. First, ATP saw the advantage of conducting descriptive case studies systematically for a broadly defined population of its projects. The approach called for flexibility in telling each project's story, and consistency in data collection across all the projects. Uniformity in data collection provided the opportunity to report select aggregate output and outcome data for the body of case studies. Parallel pursuit of detailed prospective economic case study of some of the projects in the portfolio provided an opportunity to estimate minimum net benefits for ATP 's portfolio of projects. Finally, the need for a consolidated performance measure for portfolio assessment prompted the development of CPRS, a new evaluation tool. Interlinked, the steps associated with this framework of evaluation produces a synergistic capability, well beyond the value of the steps taken piecemeal.

1.0 Developing the Project-to-Portfolio Approach

The framework builds from a case-study base to portfolio analysis, with linkages from bottom to top. This section first shows how the various layers of information are linked in this approach. Then it lists and discusses the steps for constructing the framework. Next, it assesses the evaluative outputs from applying the approach.

1.1 Multi-Tiered Analytical Capability

With reference to Figure 1, the bottom tier contains the foundational project case studies and details of all the projects in the portfolio. These case studies document the what, how, and why of completed projects. In addition to this information, for a select sub-group of the projects, detailed economic benefit estimates are provided, principally through prospective economic case study.

The second tier from the bottom of Figure 1 indicates the body of uniformly collected output, outcome, and outlook data for each project and, for a subset of projects, more extensive economic estimates of impact. These data are tied to the individual project, but can be aggregated by data category.

The third tier from the bottom of Figure 1 indicates the CPRS ratings. There is a star rating for each project, computed from weighted values of its output/outcome/outlook data.

The individual CPRS scores form a pool of CPRS scores in the fourth tier from the bottom of Figure 1. The results can be broken out by technology area, firm size, firm location, or other areas of interest as indicated in the fifth tier from the bottom of Figure 1.

Figure 1. Multi-Tiered Analytical Capability

The sixth, or top, tier contains portfolio-level information. It shows the overall distribution of CPRS ratings for all projects in the completed project portfolio. It also shows an estimate of minimum net benefits for the program, computed by aggregating benefits from the sub-group of detailed economic case studies, and subtracting total program costs from the sub-group's total benefits. The top tier also provides output and outcome data from all the case studies, aggregated by data type, such as numbers of patents or commercial products.

The resulting analytical capacity under this framework allows either a top-down or bottom-up approach. Program administrators generally start with the top level of the distribution of performance ratings and estimated minimum net benefits of the program and move down the tiers to the case studies for more detailed information. In contrast, project managers generally start with the projects they are assigned to oversee, and go up the chain to determine how their projects are performing relative to the overall program portfolio. In either case, the analytical capability of this approach facilitates managerial action to improve project and portfolio performance, and provides performance metrics to stakeholders. This ability to move up and down the various tiers makes this framework of analysis potentially useful both to higher-level program administrators and to project managers.

1.2 Steps Involved in Developing the Framework

The analytical capability depicted in Figure 1 can be developed by carrying out the steps listed in Table 1. The first step consists of several preparatory tasks:

• defining the program's major goals;
• defining the portfolio of projects for which case studies will be conducted;
• identifying the projects to receive more detailed economic case studies;
• identifying the targeted time within the projects' life cycle for conducting the case studies; and
• gaining at least a preliminary idea of the output and outcome metrics that appear best to indicate progress during the specified time period against the identified program goals and that are feasible for uniform collection.

The second step is to formulate CPRS, reexamining the indicator metrics to be used, and deciding the weights to assign them and how to combine them to signal a project's overall progress toward achieving the program's multiple goals.

The third step is to prepare a data collection template, as well as a broader case-study template, for use by case-study analysts. A template is particularly important to ensure the consistent collection of information when there are multiple case-study analysts. Consistency in the data collection is critical to implementing CPRS and satisfying other study objectives. The detailed economic case studies may also benefit from a study template or guide, but it is important to afford the analysts sufficient latitude to respond to the unique modeling requirements of each economic case study.

The fourth step is to conduct case studies for the defined population using the template. To keep evaluation cost down and to allow the cases to be carried out faster, the bulk of the case studies can be short, descriptive "mini-cases." To enable the quantitative estimation of minimum program net benefits, it will be necessary to conduct detailed economic assessments for a few projects in the portfolio.⁶

The fifth step is to apply the rating system to the projects in the portfolio using the compiled indicator metrics and the CPRS formulation as carried out in step 2. Each project then carries a star rating together with the project story and key data.

The sixth step is to calculate and depict in tabular and graphic form the distribution of project ratings for the portfolio. Distributions of performance ratings grouped by variables of special interest, such as technology area or company size, also can be calculated and depicted.

The seventh step is to compute aggregate indicator metrics for the portfolio. These metrics might include, for example, the number of publications, the number of collaborations formed, and the number of products commercialized.

The eighth step is to estimate minimum net benefits attributed to the public program. This rough estimate is derived by aggregating benefits attributed to the program for the sub-group of projects for which detailed economic case studies have been conducted and subtracting total program costs from the result. Because the portfolio examined may encompass less than the entire program portfolio, and detailed economic benefit estimates are likely only available for a fraction of the projects, the resulting estimate of program net benefits is considered a minimum value.

1.3 Resulting Evaluative Products

Carrying out the eight steps listed in Table 1 produces a number of distinct products that have potential value for project and program evaluation. As summarized in Table 2, these evaluation products are:

• a detailed descriptive and explanatory case study account of each project in the portfolio;
• a set of detailed economic cases that provide estimates of project benefits;
• aggregate statistics for the various indicator metrics;
• a CPRS rating for each project;
• the distribution of projects in the portfolio by their CPRS ratings; and
• an estimate of minimum net benefits or net losses for the entire program.

The six products are individually useful for managing the program and reporting its results to stakeholders. The linkages among the products increase their potency and usefulness as management tools.

2.0 Pros and Cons of Rooting the Approach in Case Study

ATP, like most other public R&D programs, made the case study method—both descriptive and economic—one of the mainstays of its evaluation program from the outset.⁷ There are a number of reasons for the popularity of the case study method—particularly with programs that fund scientific research.⁸ Table 3 summarizes both the advantages and disadvantages of the case study method.

2.1  Case Study Advantages

Cast in a narrative format, descriptive case studies can make complex science and technology (S&T) projects accessible and interesting to a non-scientist audience. The potential scope of the case study format is flexible and broad, ranging from brief descriptive summaries to long, detailed accounts. Using a "storytelling" approach, the evaluator may present the genesis of ideas, explore what happened and why, give an account of the human side of a project, explain goals, explore project dynamics, investigate particular phenomena, and present outcomes in their complexity without being subject to the confines inherent in most other evaluation methods. The freedom to collect multiple kinds of information makes the case study method useful for exploring ideas and constructing theories about program or project dynamics.

Pushing beyond descriptive case studies, economic case studies can combine the story­telling richness of the descriptive case study with the analytical rigor of economic analysis. Economic benefit-cost case studies typically combine qualitative and quantitative results, reflecting the difficulty of translating all important effects into monetary units.

2.2  Case Study Disadvantages

Despite its advantages, the case study method is traditionally considered to have several major shortcomings as an evaluation tool. Descriptive case studies are qualitative and anecdotal. Economic case studies, though highly quantitative, usually are successful in capturing only partial benefits in monetary units and these are usually predicated on a number of assumptions. And, whether qualitative or quantitative, case studies typically pertain to single projects, or, at best, small clusters of projects, such that their results usually cannot be generalized to the entire portfolio of projects.

2.3  Case Study, A Serious Research Tool?

Noting that distinguished scholars frequently use case study as a method of analysis, Yin, in his landmark book on case studies, asks, "If the case study method has serious weaknesses, why do investigators continue to use it?"⁹ Among the possible explanations Yin considers are that people are not trained in the use of other methods, or, for federally sponsored research, that the difficult clearance procedures required for surveys and questionnaires have made their use "a bureaucratically hazardous affair," leading researchers to favor the case study method, which is relatively unencumbered by restrictions and requirements. Yin, however, rejects these explanations for the popularity of the case study methodology. As evidence, he points out that expert analysts skilled in the use of a variety of evaluation methods use the case study method and that "federally sponsored research does not dominate the social sciences—certainly not in Europe and other countries" where the case study method is also widely practiced.

TABLE 3 - Advantages and Disadvantages of Case Study Methodology

Advantages

• Easy to understand and remember.
• Richness of detail may be useful in formulating theories and hypotheses and for understanding how and why questions.
• Good for documenting experiences, providing a benchmark for other projects, and capturing a holistic view.
• An economic case study relates project outcomes to inputs and provides quantitative measures in the language of finance.

Disadvantages

• Descriptive case study generally provides anecdotal evidence, which lacks robustness as evidence of a program's effectiveness.
• The focus is on the individual project, not on the program's portfolio of projects.
• Results for single projects and small clusters of projects generally cannot be generalized.
• Important benefits in an economic case study may be difficult or impossible to capture in monetary terms.

Identifying three types of case studies—descriptive, exploratory, and explanatory—Yin argues that the case study method qualifies as a serious research tool. He states, "In general, case studies are the preferred strategy when 'how' or 'why' questions are being posed, when the investigator has little control over events, and when the focus is on a contemporary phenomenon within some real-life context."¹⁰

Many public R&D programs, such as ATP, have numerous applications that meet these conditions. In the case of ATP, case study investigators typically are in a third-party status with no control over the contemporary developments that occur in real-life project contexts. In the case of the mini case studies, the investigators have the task of finding out and documenting how the research projects turned out and why; and why the firms turned to the government for funding. Although the survey method could be alternatively used to collect data on what happened, it is less suited for capturing data to address how and why questions.¹¹ Moreover, surveys, unlike case studies, do not capture well complex human and organizational elements of the projects, important to understanding project dynamics. In Yin's words, "... case study allows an investigation to retain the holistic and meaningful characteristics of real-life events...."¹² He continued, " 'How' and 'why' questions are likely to favor the use of case studies, experiments, or histories."¹³

3.0 ATP's Development and Use of the Project-to-Portfolio Approach

In 1996, ATP and NIST managers requested ATP's Economic Assessment Office¹⁴ to produce streamlined case studies for all completed projects, written for a lay audience, with consistent inclusion of specific information. This request was driven primarily by stakeholder requests for more information on the performance of ATP's portfolio of projects after ATP's period of funding for the projects has ended.

Prior to the development of these case studies for all completed projects, brief project descriptions prepared and made public at the time the ATP awards were announced were the only information provided consistently to the public on every project funded. Survey data provided information for all or most projects, but the data were released only in the aggregate in order to protect proprietary project or company information. Workshops and conferences provided the public with information on many projects, but not in a written and consistent format that was widely available, and generally not for projects in the post-completion period. Economic case studies focused on a single or small cluster of projects; econometric case studies used specialized databases in ways that did not reveal in-depth information about individual projects.

The new case study product was added to a rich array of existing evaluation methods that provided substantial valuable information about ATP's performance. Results from surveys, for example, showed what percentage of the project portfolio had published, patented, sold products, collaborated, attracted funding from other sources, experienced employment growth, and other interim achievements that were evidence that ATP was meeting its mission objectives. Econometric studies showed, among other things, that ATP was selecting projects with larger than average spillover potential, providing a halo effect as award-recipient firms were more successful than non-recipient firms in attracting additional private sector funding, and having a positive effect on the rate of patenting by award recipients. Detailed benefit-cost studies provided estimates of returns on investment for selected projects. Descriptive case studies for selected projects provided details about those projects and explanations about ATP's role. A variety of analytical and conceptual modeling studies advanced fundamental understanding of the program and its operations. Indicator data signaled that projects were making progress in a number of areas. The new case study product provided additional capability to ATP's evaluation program.

3.1 Mini Case Studies Using Uniform Data Collection for All Completed Projects

ATP's Economic Assessment Office responded to the request for a new evaluation product by commissioning descriptive mini case studies with uniform data collection for all completed ATP projects, written approximately 3 to 5 years after project completion. There were several reasons for setting the cases 3 to 5 years after project completion. First, because of ATP's relatively short history; few projects had progressed far enough at the time the mini case study effort was begun. Second, fixing the case studies 3 to 5 years after project completion had the advantages of yielding a manageable number of projects to get the effort underway, allowing adequate time for companies to make post-project progress, and avoiding problems associated with attempting to obtain detailed information long after a project has been completed.

The mini case studies told the project story, describing what was done, how and by whom the project goals were accomplished, the role of ATP, and assessed the outlook for further developments. The mini case studies focused on the original award recipients, but branched out to include licensees and takeover companies if these organizations provided the focus of current activity. In a few instances and to a limited extent, the mini case studies followed the mobility of people.

Analysis of ATP's mission and program goals, and specifically requested information by NIST and ATP administrators, guided the identification of data to be uniformly collected for all the completed projects. The specific data collected are listed and discussed further below under the section, "Defining Mission-Driven Goals for CPRS Development."

3.2 Publication of ATP's First Status Report

There were 38 projects in the first group for which mini case studies were developed. These projects were collectively published in a report.¹⁵ The project story was given in 4-5 pages of text and key data were highlighted for each mini case study. An overview chapter reported on aggregate output and outcome data for the portfolio of completed projects, and provided an estimate of minimum net benefits from the program. ATP called these mini case studies, "Status Reports," to underscore the fact that they provided a snapshot view of project accomplishments at a specified time, and that subsequent developments were likely to occur.

The first volume of Status Reports proved useful, but it did not provide program administrators a composite measure of portfolio performance. Policymakers and other stakeholders were interested in the overall performance of the program and ATP managers were interested in what aspects of the portfolio were performing well and which ones were falling short. With the program's multiple goals and multiple output and outcome measures, it was difficult to get a clear reading of performance across the portfolio, even from the aggregate indicator metrics. Some projects scored high in one outcome area; and low in another. Program administrators were unable to interpret the results in terms of overall portfolio performance. A pressing question of policy makers could not be answered using the evaluation tools at hand, "How are projects in ATP's portfolio performing overall against ATP's mission-driven objectives in the intermediate period after project completion and before long-term benefits are realized?" And more specifically, "What percentage of the portfolio is made up of strong performers, weak performers, and those showing a moderate level of performance?" Thus, the first volume of Status Reports, while valuable from the standpoint of reporting on the post-ATP performance of individual projects, could not be used as a management tool for addressing portfolio performance.

3.3 New Features in ATP's Second Status Report

To address portfolio performance, an easy-to-grasp overall performance measure for each project needed to be added to the richness of the individual project details so as to offer a means of comparison across the completed projects examined.

In 2001 a replacement volume containing status reports for the first 50 completed projects was published.¹⁶ Like the first volume, it contained an overview chapter with aggregate indicator metrics and estimated minimum net benefits from the program's investment to date. In addition to the earlier volume, it contained a new feature: Composite Performance Rating System (CPRS) ratings. Each project was assigned a 0 to 4 star rating, where a low star rating indicated weak performance, and a high star rating indicated strong performance against ATP mission objectives. The overview chapter showed the distribution of projects by their composite performance. An appendix listed the 50 projects in terms of their individual CPRS ratings. Another extension provided by the second volume was inclusion of patent citation trees to illustrate visually knowledge spillovers from the project.

The latest batch of 50 mini case studies for completed projects is now available. A data template is being used to ensure that data for the new cases are consistent with the first 50 projects. The newly assessed projects carry CPRS ratings, and the distribution of projects by performance has been computed for the larger group. The entire portfolio of ATP Status Reports of completed projects are available on ATP's web site with a searchable feature that facilitates retrieval of status reports by state, technology area, CPRS rating, size of company (for project lead), and other characteristics of the mini case studies.¹⁷

5. ATP, like other agencies, included select indicator metrics in its performance reports under the Government Performance and Results Act of 1993 (GPRA), which requires federal agencies to develop strategic plans, relate budgetary requests to specific outcome goals, measure performance, and report on the degree to which goals are met. An overview of the GPRA is provided in Appendix 1 of the U.S. General Accounting Office's Executive Guide, Effectively Implementing the Government Performance and Results Act (GAO 1996).

6. Specification of the selection plan for the detailed economic case studies is beyond the scope of this paper. In the example used in this paper, the cases were selected from among what were expected to be relatively strong projects—an effective way of estimating minimum net portfolio benefits, given the distribution of projects by performance.

7. A recent benchmarking workshop among five S&T programs in the U.S., and several S&T programs in other countries revealed that they all used case study as one of their main evaluation methodologies. See Ruegg (2003). Program administrators from the U.S. National Science Foundation, National Institutes of Health, the Department of Energy's Office of Science and Office of Energy Efficiency and Renewable Energy, the National Institute of Standards and Technology's Advanced Technology Program, Finland's Tekes Program, Canada's Industrial Research Assistance Program, and Israel's MAGNET Program all noted the importance of case study methodology to their evaluation efforts.

8. Branch et al., (2001).

9. Yin (1994).

10. Yin (1994), p. 1.

11. ATP used the case study method to capture all the output data for the first 50 completed projects, but now is positioned to use a combination of surveys and case studies to collect the output or "what" data. The case study method continues to be used to capture the 'how' and 'why' of project developments.

12. Yin (1994), p. 3.

13. Yin (1994), p. 3.

14. ATP's Economic Assessment Office (EAO) is comprised of a staff of economists, statisticians, and information specialists that organizes and manages evaluation activities for ATP, conducts studies in- house, and commissions studies with universities, consulting firms, other assessment organizations, and
with individual economists, in part through the National Bureau of Economic Research (NBER).

15. Long (1999).

16. ATP (2001). This volume contained the first 38 status reports plus 12 new ones, in addition to the CPRS (Composite Performance Rating System) ratings.

17. The searchable web site is available at www.atp.nist.gov, or specifically at http://statusreports.atp.nist.gov/

Return to Table of Contents or go to next section.

Date created: June 1, 2006
Last updated: July 3, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)