A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 6:  Case Study Method

ATP has used case studies for multiple purposes throughout its first decade. Case studies have helped make the technical and economic aspects of its complex technology projects more accessible to non-scientists. Case studies have helped explore the genesis of projects and programs, and tell the stories of the people and organizations behind the projects. Case studies have helped answer why and how questions, explain roles and goals, investigate project dynamics, track progress, identify market applications, measure outcomes, and—performed in multiples—estimate portfolio performance.

All of the case studies presented in this chapter describe subject projects, organizations, and technologies. Reflecting ATP’s emphasis on demonstrating economic impact, many of them include substantial quantitative elements, including projections of social rates of return. Table 6–1 lists a selection of ATP’s case studies on which this chapter is based. The main objectives and key features of the studies are shown.

This chapter is organized according to three major case-study objectives: modeling and estimating economic impacts, estimating project and portfolio performance using multiple cases and progress indicators, and explicating selected program features and exploring dynamics. The chapter emphasizes the approaches and models used to carry out the case studies; findings are presented only as they contribute to a fuller explanation of the underlying models or cases.

Economic Case Study of Individual Projects

The case studies illustrate the point made earlier that evaluation is an art as well as a science and a craft. Each researcher or team of researchers takes a somewhat different approach to creating a case study, depending on project particulars—objective timing of the study, market applications, data availability—as well as their research expertise, study budget, and research perspective. The first group of case studies provides economic impact  stimates for individual projects funded by ATP. These cases also provide descriptions of the technologies and the innovating organizations, the sources  of economic benefit, and the role of ATP in the projects. The first approach reported here was developed for application early both in ATP’s history and in the life cycle of the surveyed projects. Lacking market observations on economic benefits at the time the studies were undertaken, the researcher took
a novel approach to estimating minimum project benefits based on research cost savings. The second approach covered seven tissue-engineering projects, all aimed at providing improved medical treatment at lower cost. The team of researchers developed a method for estimating expected social economic return on public investment, using concepts from health care assessment to measure patient quality-of-life benefits in several of the cases. The third approach listed combined microeconomic estimation techniques with macroeconomic modeling to estimate national benefits from adoption of new automotive technology. The fourth approach focused greater attention on market research as a way to explore multiple target applications for the technology, emphasized combining estimated effects from multiple benefit streams, presented an explicit treatment of qualitative benefits in conjunction with quantitative estimates, and provided a more transparent exposition.

Table 6–1. Ten of Sixteen Studies Featuring Case Study Represented*

* Note: Other studies using the case study method as a secondary method listed in Table 3–4, but not explicitly treated here, are by Austin and Macauley, 2000; Przbylinski, 2000 draft; Fogarty et al., 2000 draft; Liebeskind, 2000 draft; and Dyer and Powell, 2002.

Each of these studies is unique, raising the standard evaluation question of how to generalize findings from case studies. Adding weight to this reservation is that, aside from the set of tissue engineering studies, each was a separate undertaking commissioned at a different time, and no attempt was made to establish a uniform set of questions, data collection procedures, or other uniform research protocols among the studies by different researchers. This decentralized approach was considered appropriate at the time the studies were commissioned as ATP was purposefully experimenting with different approaches and testing the analytical capabilities of contractors. Each of these studies was directed at a somewhat different question and in a different set of circumstances. Each of the studies represents a legitimate approach to the particular case and set of problems it tackled; in the aggregate, they highlight the flexibility as well as strengths and weaknesses of case studies within a larger portfolio of evaluation techniques. ¹⁴⁸ The key features of each of this first group of studies are summarized in turn.

Estimating Minimum Benefits of New Technology in Terms of Research Cost Savings and Competitive Improvements

In its first competition in 1990, ATP funded five joint ventures among a total of 11 projects selected—several of them relatively large, five-year efforts. Eager to learn more about how these early joint ventures were performing and to document results, ATP commissioned Albert Link, University of North Carolina- Greensboro, to evaluate three joint ventures near the midpoints of their five-year duration. ¹⁴⁹ Each of the three projects represented an attempt by a group of U.S. companies within an industry sector—the printed wiring board (PWB) industry, the data storage industry, and the advanced display industry—to respond to foreign competition and enhance their industry’s competitiveness through the development of a suite of “leap-frog” technologies. Subsequently, ATP focused on evaluation of the PWB project, and sponsored Link to update the analysis of the joint venture at project end. ¹⁵⁰

Link’s Approach

Link used essentially the same approach in each of the case studies he performed: He described the industry, the technology, the nature of the collaboration, the major research tasks, the project’s organizational structure, and the role of ATP. He also identified changes in the participating organizations and research plan as the project unfolded. Because of the early stage of the projects he investigated, Link focused on quantifying research cost savings from the collaboration and on changes in competitiveness, rather than on attempting to forecast benefits from the technology in use. In each case, he surveyed the participants to collect data needed to estimate impacts.

By examining the characteristics of the member companies, Link assessed the nature of the collaboration. For example, although the PWB joint venture is primarily a horizontal collaborative research arrangement, Link found that the members were actually not head-to-head competitors. This finding was important because it helped explain why the joint venture members collaborated more fully and shared their research results more extensively than in cases where joint venture members were direct competitors. ¹⁵¹

Data Collection

To collect data for the quantitative part of the case study, Link used a survey with three parts, and one with a counterfactual element. The survey part of the PWB case study was described in earlier in Chapter 5.

Link collected additional data by asking members of the project’s steering committee—a management group made up of representatives from the participating organizations—to respond, in terms of the level of agreement/disagreement, to a set of ten statements. The statements describe the importance of the PWB joint venture to the company and the industry in terms of ability to refine manufacturing technologies and commercialize new scientific discoveries and technologies more rapidly and to improve competitive position.

Link collected qualitative information from members of the steering committee, who were asked to complete the following statement: “My company has benefited from its involvement in the PWB joint venture in such non-technical ways as ...”. Also, they were asked to listen to a reading of the goals of ATP, and to indicate in response the degree to which they thought ATP goals had been fulfilled in their project.

The remaining data Link collected in support of the impact analysis related to effects from using the resulting technology. He asked members of the steering committee to estimate their own company’s productivity gains traceable to using project outputs in their production. These data were sparse because the research was just concluding at the time of the study.

Presentation and Interpretation of Results

Link used the collected data to provide estimates of minimum impact for the PWB joint venture, part of which are quantitative and part qualitative. To obtain an estimated minimum dollar value of the assistance provided by ATP, Link combined the various cost savings from efficiency gains in carrying out the projects as a joint venture. He counted costs savings only for that part of the research the companies said they would have pursued on their own without ATP, because otherwise they presumably would have incurred no research costs.

Table 6–2 summarizes direct impacts to member companies: research cost savings (a total of $35.5 million at project end), production cost savings ($5.0 million at project end), and indirect impacts on member companies (that is, increase in competitive position in world markets). It also shows partial spillovers to the PWB industry: 214 papers, 96 conferences, and increased competitive position for the U.S. industry as a whole. For comparison, the table brings forward the summary results of the earlier case study of the PWB project that Link performed two years into its five-year timeframe.

In addition to the results summarized in the table, Link pointed out potential value in the new capabilities the companies now have due to approximately half of the total of 62 project research tasks that were omitted from the cost savings calculation. ¹⁵² He also noted reduced cycle times for new project and process development and the presence of substantial technology transfer products providing pathways for the rest of the industry to benefit from the projects outputs. As is often the practice in case studies, Link included representative anecdotal responses from the companies about how they have benefited.

Link presented the results as “a conservative lower-bound estimate of the longrun economic benefits,” and as “partial and preliminary estimates of project impacts.” He pointed out that the bulk of production cost savings and performance gains would be realized in the future as the technology results diffuse and are more widely implemented.

Table 6–2. Summary of Survey Findings on Partial Early-Stage Economic Impacts

* These impacts are based only on those research tasks that the members thought they would eventually have done without ATP, and not the cost and time savings associated with the new capabilities resulting from those tasks that they would not have done at all without ATP.

Source: Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. 34.

Modeling Private and Social Benefits of a Set of Related Medical Technologies

Among the more ambitious and methodologically important case studies commissioned by ATP over its first decade was that conducted by economists at Research Triangle Institute (RTI) to estimate the economic impacts of a portfolio of seven ATP-funded projects in medical technology. ¹⁵³ The study is valuable for several reasons. As an approach to portfolio assessment, it illustrated how use of a common, consistent methodology across a set of technologies within the same industry can be used to identify “project” or “technology” characteristics that affect the relative economic impacts of these projects. The study also showed how a formal model of relationships can be used to guide collection of information and data, and is notable for the care and detail with which it assessed ATP’s programmatic objectives in the context of specific technologies. Finally, the study has value because, among those presented, it most explicitly links its design to the central analytical models and concepts used to articulate ATP’s mission, while at the same time addressing “specific methodological challenges that have not been addressed in ATP’s previous methodological development efforts.” ¹⁵⁴

Study Objectives

The study had three objectives: First, to develop a methodology for estimating the expected social rate of return on public investment in ATP-funded projects with medical applications; second, to apply the model to all of the ATP-funded multiple-application tissue engineering projects funded by ATP between 1990 and 1996; and third, to estimate the composite social return and compare it with the composite private return for the set of cases. As shown in Table 6–3, four cases were performed in greater depth than the others.

Table 6–3. Overview of ATP Projects Included in this Study

Note: Tissue engineering produces materials that can be used either to replace or correct poorly functioning components in humans or animals. Throughout this report we refer to each project by the abbreviated title listed below the full title.

* BioHybrid was approved for a 2-year no cost project extension.

Source: Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–13.

RTI’s Approach to Estimating Benefits and Costs

RTI’s approach was to build on Mansfield’s model for estimating private and social rates of return, modifying it to take into account the specific forms of benefits generated by medical technologies. It also incorporated the evaluation and policy design precept implicit in Mansfield’s work and made explicit by Jaffe: that because private sector R&D tends to generate social rates of return, the test of ATP’s economic impacts are the social rates of return it generates above those likely to have resulted from private sector activities alone.

RTI modeled ATP funding of R&D projects as affecting the development of medical technology in three ways: (1) accelerating the technology’s benefits (i.e., bringing benefits to the private sector, patients, and society sooner and for a greater number of years than without ATP funding); (2) increasing the likelihood of success (i.e., increasing the amount of R&D conducted and thereby the likelihood that a project will be technically successful); and (3) widening the scope of the project and enabling the company to apply its technology to additional diseases or patient populations. Figure 6–1 illustrates the model underlying the selection of relationships and variables for which information and data were collected.

Figure 6–1. Elements Determining Social Return on Public Investment and Social Return on Investment

Estimation of each of the generic effects, in turn, represented construction of a set of scenarios detailing what was expected to happen because of ATP funding relative to what would have happened in the absence of the funding. Thus, for example, if it were assumed that ATP funding accelerated bringing a product to market by two years, the model assumes that the with-ATP innovation starts generating benefits two years earlier and has an economic life two years longer (and thus a higher net present value) than the same innovation produced without ATP funding.

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–5.

Table 6–4 relates these differential benefits across tissue engineering projects to the effects of ATP funding. The single greatest source of differential effects was estimated to be acceleration by ATP of the rate at which a technology is brought to a marketable stage. Company officials involved in developing biopolymers for tissue repair, in RTI’s words, reported that without ATP assistance the company might not have developed this technology at all or might have developed it so slowly that the market opportunity for it would have passed before it was ready for commercialization. In this case, the study assigned a 10-year advantage in estimating project benefits with ATP support.

Table 6–4. Impact of ATP Funding on the Development of  Medical Technologies for Seven Tissue Engineering Projects

Note: Our model allows conceptually for ATP funding to widen the scope of a project. In practice, for the applications in this study, there was little or no impact in all but two cases, which we did not quantify. *This is the number of years of acceleration reported by the ATP-funded companies. For the one to two year ranges, we used the lower number for our analysis. For the three to five year range, we used the midpoint of the range.

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–23.

RTI modified the Bass diffusion model ¹⁵⁵ to estimate adoption of the new technologies. The rate of adoption was increased during the earlier period and decreased as the market potential was approached. RTI assumed that a newer ¹⁵⁵ Frank M. Bass, “A New Product Growth Model for Consumer Durables,” Management Sciences , 15(5):215—227, 1969. technology would completely supersede each of the ATP-funded technologies after a 10-year period and models a cessation of diffusion at that time.

RTI separated net benefits estimation into those occurring in the medical technology sector and in the health care delivery sector. For the medical technology sector, net benefits included estimated change in revenues from sales of the new medical products and procedures, less investment and production costs incurred in bringing them to market, as compared with the displaced defender products and procedures; that is, the change in profits from having the new technologies. For the health care delivery sector, net benefits included reductions in the costs of health care and the value of increased health benefits to patients.

To estimate the value of health benefits, RTI adopted a concept called Quality Adjusted Life Year (QALY), developed in the field of healthcare to allow quantification of health changes in terms of the quantity and quality of life. ¹⁵⁶

Where a year of life at full health is assigned a QALY value of 1.0, and death is assigned a value of 0.0, in between states are assigned QALY values between 0.0 and 1.0. ¹⁵⁷ The QALYs must be translated into dollar values. The steps required in valuing per-patient changes in health outcomes and RTI’s methodological approach at each step are summarized in Figure 6–2.

Figure 6–2. Valuing Per-Patient Changes in Health Outcomes

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–9.

As noted, one of the contributions of the RTI methodology is that it offered insights into the sources of variation in rates of return across a portfolio of similarly directed technologies. As stated in the report, “Social returns to these projects can vary with respect to the number of patients treated, the value of the health benefits of the new technology, their impact on health care costs, and the probability of technical success.” ¹⁵⁸

Data and Assumptions

Information on market potential, R&D expenditures, benefits to patients, and other variables necessary to compute social and private rates of return for each case was collected from a number of sources, including representatives from the companies receiving ATP funding. According to Martin et al.:

The most important sources of information about each technology were representatives of the companies receiving ATP funding. We interviewed representatives of each lead company and, in some cases, also interviewed representatives of partner companies. We also talked with a number of physicians and consulted a variety of secondary data sources, including medical literature and statistical databases, to develop estimates of costs and benefits. (p. 1–14)

The inclusion of estimated benefits from health improvements was dependent on the researchers being able to find existing QALY values, because estimating them was beyond the scope of the project. These values have been developed for certain health conditions and diseases from surveys of affected populations, such as cancer patients and diabetics, based on choices expressed by respondents; however, they are not available for every disease or condition. Where they found suitable QALY data, the researchers used the data to develop benefits from improved health outcomes. For example, the researchers found much of the data required for the model of health outcomes related to new treatments for diabetes from the Diabetes Control and Complication Trial (DCCT). ¹⁵⁹ They found, for example, that blindness from retinopathy carries a QALY of 0.69; end-stage renal disease carries a QALY of 0.61; and lower extremity amputation, a QALY of 0.80. For illustration, Table 6–5 lists various QALYs for different health states and corresponding study source.

Table 6–5. Comparison of QALY Utility-Weights for Different Health States

* For biopolymers, the two sets of figures are identical because all of the social return can be attributed to ATP investment.
**See notes to Table 6.5 in the original for an explanation of the derivation of the composite measure of return.

Source: Excerpted from RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 2–23.

To determine the dollar value of the change in the patient’s well being, RTI researchers estimated the economic value of a QALY based on willingness-to-pay values for avoiding illness and accidents taken from existing studies. ^{160, 161} They also drew probability data from existing studies, such as the probability of blindness given diabetes from the DCCT study.

Composite Private and Social Rates of Return

Table 6–6 summarizes the study’s estimated expected social return on total investment and the expected social rate of return on public (ATP) investment for each of the ATP projects examined in the RTI study. It also shows the composite rate for all the projects taken together.

Table 6–6. Social Return on Investment and Social Return on Public Investment: ATP Projects in Tissue Engineering for a Single Preliminary Application

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies , 1998, p. 1–22.

Based on these results, the authors concluded:

ATP funding is responsible for inducing about 31% of the total social returns from all of these projects over 20 years. For the individual projects, the effect of ATP on social returns ranges from about 25% to 100% of the social returns. (p. 1–22)

Table 6–7 reports the composite private return on investment for the seven projects. Based on comparing the social and public returns in Figure 6–7 and the private returns in Figure 6–8, the authors concluded:

The wide disparity between social and private returns indicates the importance of ATP incentives to the private sector to pursue these technologies. Because the social returns far outweigh the returns to the companies developing, commercializing, and producing these high-risk projects, the private sector may under invest in these kinds of high-risk projects. (p. 1–24)

Table 6–7. Composite Private Returns: ATP Projects in Tissue Engineering for a Single Preliminary Application

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–24.

Limitations

As careful, systematic, and methodologically focused as the RTI study was, it still has limitations. Paramount among these, as with many of the case studies reviewed in this section, is that it is a projection of expected net economic benefits, not a measurement of observed benefits. None of the tissue engineering technologies covered in the RTI study had entered commercial use at the time of the study, although some were in clinical trials. In fact, at the time of the study, it had not yet been demonstrated fully that all would function technically as expected, thereby compounding uncertainty in the estimated outcome. Thus, there was a “shortage of ex post empirical data.” ¹⁶² This limitation, clearly, was a function of the time at which the case study was done, and not a function of the case study method or the implementation of this case.

Another limitation of data rather than the model is the fact that the study only estimated patient benefits from improved health outcomes when there were preexisting QALY data for the relevant medical conditions. Thus, the disparity in the size of net benefit estimates among the projects to some extent reflected the inclusion of patient health care cost, but not of patient health outcomes in several cases. Exercising the model for only one application of multi-use technologies is a choice reflective of budget limitations rather than a shortcoming of the model.

What most distinguishes this study is its explicit attention to methodological development; linkage to the multiple, attributed impacts of ATP funding, and formal ties to core theoretical constructs. Further, by analyzing all the projects funded by ATP within a single technological area, the study strengthened its ability to generalize results within that area.

Estimating Market-Based Economic Impacts from Automotive Technology Combining Microeconomic and Macroeconomic Modeling

This section pairs two case studies: CONSAD Research Corporation’s case study of dimensional control technology, the “2mm Project,” ¹⁶³ and former economist at the National Institute of Standards and Technology (NIST) Mark Ehlen’s case study of flow-control machining technology. ¹⁶⁴ Although performed by different researchers, these two cases bear similarities. In both cases the first application of the multi-application technologies was to automobiles. Both entailed primarily vertically structured joint ventures, which bring together in some role supplier-innovators, universities, and large automobile assemblers. Both dealt with new manufacturing process technologies that offered quality/performance improvements. Both employed a micro-level examination of impacts arising from the technical characteristics of the project. Both attempted to link microeconomic modeling of firm- and industry-level impacts to macroeconomic modeling of national economic impacts.

The case studies have several important differences. The first case study was done on a much smaller budget, a shorter schedule, at an earlier time, on more of an experimental basis, and with less detail than the second. The first case study uses two, largely disconnected approaches: a microeconomic approach to estimate production and maintenance cost savings based on unit savings and current production volumes, and a macroeconomic approach to estimate total industrial output and employment changes, based on expert judgment about the increase in sales of U.S.-made vehicles due to technology-based quality improvements. In contrast, the second case study systematically built its model from firm level, to industry level, to the national level, integrating across the micro- and macro-parts of the analysis. Features of these two cases are discussed and compared below.

Establishing Impact Expectations by Examining Technical Characteristics

Both of the case studies explained to the reader why and how the projects’ technical accomplishments could logically be expected to yield benefits, an important part of building the case story. In the case of the 2mm technology, CONSAD researchers explained that U.S. auto assembly plants required a cost-effective method of reducing dimensional variation in auto body assembly, using the existing workforce. The project developed a new metrology-based process for improving the fit of discrete manufactured parts, with potential application to multiple manufacturing industries. Four types of direct benefits were expected from its application to automobile manufacturing: (1) decreased production costs, (2) decreased product maintenance costs, (3) improved product quality, and (4) reduced time required to launch new products or product models. Experimental implementation of the technology in five U.S. auto assembly plants at the time of the study provided CONSAD with estimates of unit cost reductions.

In the flow-control machining project, Ehlen explained how the multi-application technology increases the functional precision of cast-metal parts that carry fluids in interior passageways. Applied to auto engines, the improved precision can increase engine horsepower, increase fuel efficiency, reduce emissions, and reduce engine costs. Ehlen provided diagrams showing how the efficiency of combustion is improved. As in the previous case, performance was informed by actual data—in this case, from testing a prototype-working machine on engine manifolds. Ehlen related how the improved technical capabilities could potentially be deployed in the auto industry in alternative ways, affecting the resulting benefits. For example, in the face of fuel shortage, it could be applied in producing engines for all vehicles to decrease fuel consumption across the board. It could be used to meet increased Corporate Average Fuel Economy (CAFE) requirements. It could be used in some vehicle lines in order to sell other, less fuel efficient models, while still meeting overall the existing CAFE requirements. It could be used in specialty vehicles to increase horsepower. In other words, there are a variety of possible strategies for deploying the new technology.

Investigating the Role of ATP

Both of the case studies addressed the role of ATP as they considered why federal assistance was needed, particularly given that large end-user companies were present in both joint ventures. They came to much the same multi-reason conclusion about why the project would not likely have gone forward without ATP involvement. As Ehlen writes:

Ford and GM are unlikely to unilaterally adopt a new process that has not been proven to work; the FCM [flow-control machining] processes are particularly challenging since both constitute a radical departure in finishing processes—manufacturing directly to functional performance.  Ford and GM are also unlikely to directly collaborate on a new process, since they are direct competitors on routine business matters and have concerns about federal antitrust-law enforcement. They tend not to fund the research of their suppliers. The suppliers would not perform the research themselves; they generally do not have the capital to do extensive in-house research—particularly not high-risk research. University researchers are typically interested in doing their own research, not the research of a supplier to automakers, and are not able to self-fund the type of research. (p. 4)

CONSAD researchers emphasized the difficulties of achieving cooperation among industrial participants who frequently compete against one another, or forging a joint research undertaking among different members who “might realistically expect notably different returns from their involvement in the project.” ¹⁶⁵ In the judgment of the authors:

It appears unlikely that (a) this complex joint venture could have been formed and (b) funding for the research project could have been coordinated without direct administrative and financial involvement by the federal government. (p. 10)

Modeling the Technology’s Adoption by Auto Manufacturers

The two case studies differed considerably in their modeling of the take-up of each technology. Based on reported steady adoption of the 2mm technology by a growing number of assembly plants at the time of the study, CONSAD researchers assumed successful commercialization within the automobile manufacturing industry within a relatively short period of time.

In contrast, Ehlen included as a major part of his study the assessment of the likelihood that the automobile industry would implement the flow-control machining processes, outlining two implementation paths for estimating nearterm and longer-term impacts. He used historical adoption models of similar fuel-efficiency enhancement by auto manufacturers in modeling adoption of the new processes. Figure 6–3 illustrates the adoption modeling. The horizontal portion of the heavy solid line shows the near-term conservative view that the processes would be adopted at an introductory level only, maintained for five years, and then dropped. The upward sloping portion of the line indicates the longer-term, more optimistic projection of a broader implementation at the historical adoption rate of fuel injection technologies, implemented over 20 years by 80% of the market.

Figure 6–3. Historical Adoption Rates of Technologies Enhancing Fuel Efficiency, and Implementation Rates used by Ehlen in the Case Study

Source: Ehlen, Economic Impacts of Flow-Control Machining Technology: Early Applications in the Automobile Industry, 1999, p. 54.

Data and Assumptions

Both case studies were limited in their assessment by the recentness of the technological innovation, and the absence of market-based data. As stated by CONSAD researchers:

Because the technologies developed by the 2mm Project are new, their impacts on industrial production and economic activity are not yet revealed in the extant empirical data on industrial performance. (p. 17)

In the absence of market data, the CONSAD team turned to two expert panels for estimates of the magnitudes of impacts. The first group of experts was composed of individuals knowledgeable about the substance of the technologies and their likely impacts on costs and quality. The individuals interviewed were primarily university researchers, manufacturing engineers, and technicians and engineers involved with the initial implementation of project results at five automobile assembly plants. The second group was composed of individuals knowledgeable about the industries and markets in which the technologies would likely be used; these experts were asked for their assessments about the expected extent and rate of adoption of the technologies in specific industries and markets.

Limiting validation of the work of these two panels was the lack of detail provided due to concerns about confidentiality. “The individual sources of information and judgments, and information for individual plants and firms adopting technologies that have resulted from the 2mm Project are not cited because of the proprietary and confidential nature of the data about current and expected cost savings and expected product demands.” ¹⁶⁶ Similarly omitted in the study’s report was the means by which the judgments of the two panels were put together. “The plausibility of the judgments provided by the two groups of experts has then been evaluated by examining the coherence among the judgments provided by the various experts in each group.” ¹⁶⁷ Thus, the CONSAD study lacked transparency.

Ehlen seems to have faced fewer obstacles in obtaining and citing industry and firm data due to company confidentially. In general, the data, assumptions, and step-by-step procedure are more transparent in Ehlen’s study. Ehlen received close cooperation, particularly from the major innovator, Extrude Hone, who took a keen interest in the case study and seemed unusually willing to share data. CONSAD also received close cooperation from the companies in the joint venture it studied, but apparently faced more restrictions on the publication of data. Without cooperation from project participants and the ability to attribute data to sources, a researcher will have a difficult time conducting a detailed and replicable case study.

Using Macroeconomic Modeling in the Case Study

Both studies used macroeconomic modeling to estimate national impacts from using the technology in the auto industry. In fact, the major methodological fillip to these studies relative to other ATP case studies was the effort to scale up economic impacts through the use of a macroeconomic inter-industry model. They both used the REMI (Regional Economic Modeling, Inc.) model for this purpose.

The application of REMI in these two projects defined the limit of ATP’s use of macro-economic modeling as an adjunct to case study over its first decade of evaluation. Attempting to use macroeconomic modeling to assess the impact of a project, or even an entire program, is controversial. The “noise” in a $10 trillion economy is likely to overwhelm the measures of a macroeconomic model of the U.S. economy. Yet, the REMI model, comprised as it is of regional components and a set of structural equations linking inputs and outputs, prices, and consumer spending, offers the possibility of estimating project impact at the national level, provided the subject technology will have sufficient impact to show up at an industry-wide level and can be effectively captured in the model’s variables and causal linkages. In both the case studies treated here, it was thought that the extensive participation of large auto manufacturers provided conditions that would allow REMI modeling to be used. But for most ATP projects it is unlikely that necessary conditions would be met, and a macroeconomic model would not be an appropriate evaluation tool.

The CONSAD study applied the REMI Economic and Demographic Forecasting and Simulation 53-Sector (EDFS–53) model in conjunction with analysis based on the input-output (I-O) tables of the U.S. Department of Commerce’s Bureau of Economic Analysis. The model was used to estimate changes in industrial production and employment due to the projected increase in autos resulting from an increased combined market share of the participating U.S. auto manufacturers, based on expert opinion about the change in demand for U.S. assembled autos due to improved quality.

In contrast, Ehlen used a more detailed REMI model, and systematically built and integrated from the microeconomic modeling to the macroeconomic modeling. First, he estimated the impact on firms of near-term implementation over a fiveyear implementation path. Next, he estimated changes in industry performance and the change in annual sales for the three industry sectors involved in the supply of the technology. Finally, he used market quantities in the REMI analysis to estimate macroeconomic impacts. Table 6–8 summarizes the REMI findings for the year 2004, based on the assumed five-year, conservative implementation path.

Table 6–8. Annual Impact on U.S. Macroeconomy of Near-Term, Five-Year Implementation Path: Year 2004

Note:  Dollar concepts are in constant dollars.

Source: Ehlen, Economic Impacts of Flow-Control Machining Technology: Early Applications in the Automobile Industry, 1999, p. 46.

Estimating Net Benefits from Multiple Applications of an Advanced Refrigeration Technology

Whereas the case studies presented in the two preceding sections each performed a benefit-cost analysis for the single most promising application of the technology, the case study presented in this section investigated multiple applications. Prepared by Thomas Pelsoci, managing director of Delta Research Company, the case study examined closed-cycle air refrigeration technology (CCAR), funded by ATP in 1995. ¹⁶⁸ The joint venture project was completed in 1999, and, after subsequent corporate product development efforts, yielded “a cost-effective system for delivering ultra-cold refrigeration in the -70ºF to -150ºF temperature range to food processing, volatile organic compound, and liquid natural gas applications.” ¹⁶⁹ The system uses environmentally benign dry air as the working fluid to replace harmful refrigerants.

This study has several features that make it a good example of an economic case study. It has a clear technical characterization of the technology and its state of development; an assessment of the functional capability of the technology; an analysis of potential markets; description of pathways to commercializing in those markets; an assessment of market demand; a straight-forward, transparent benefit-cost analysis with clear identification of data and assumptions; discussion of the counterfactual; estimation of both private and social benefits; and inclusion of qualitative benefits.

Attention to Test and Demonstration Results

Given the prospective approach of the benefit-cost analysis, the attention the study gave to results of tests and demonstration of CCAR in operation takes on added importance. When technical feasibility, in addition to market feasibility, is in question—as it was in several of the tissue engineering case studies examined earlier—project risk is substantially increased. ¹⁷⁰ To address the question of CCAR’s technical feasibility, ¹⁷¹ Pelsoci cited the conclusion of project participants that “CCAR met or exceeded all acceptance criteria and successfully demonstrated its technical feasibility.” Thus, the technology has been demonstrated to work, freeing the researcher to focus on the question of whether it will be adopted, when, and for what uses.

Market Assessment

The market analysis emphasized fact-finding and analysis of both primary and secondary markets for the technology. CCAR was termed a niche technology because it represented a cost-effective alternative only within the specified temperature range. Mechanical refrigeration provides cooling above the -70ºF range. Cryogenic refrigeration provides cooling below -70ºF, but its high cost may limit industrial applications.

In the U.S. food industry value chain, the study identified “further-processed foods,” a $131 billion market, as the targeted primary end market for the CCAR technology. As illustrated in Figure 6–4, the study identified key market drivers of this market segment.

Figure 6–4. Market Drivers for the Further-Processed Food Industry

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 12.

The study related each of the market drivers to changing demographics. It explained how colder freezing is linked to more rapid freezing and in turn to higher quality, and how the CCAR technology provides an enabling technology for meeting market demands in the targeted primary market segment. It sourced two existing market studies by independent market research companies to assess the level of interest among food companies for the CCAR technology. It also relied on information gleaned from discussions with expert technical and sales staff at the joint venture companies, food industry associations, and food companies.

The study identified five promising pathways for marketing CCAR refrigeration services for food processing based on primary research and analysis completed during 2000 and early 2001: (1) replacing liquid nitrogen as a refrigerant, (2) replacing carbon dioxide as a refrigerant, (3) installing CCAR units at plants with expanding production, (4) installing CCAR units at newly constructed food plants, and (5) exporting into the overseas market. The study identified the four potential secondary markets for the CCAR technology shown in Table 6–9, and discussed the opportunities and barriers in these markets and explored the pathways to commercial acceptance. The study concludes that the residential, automotive, and other warmer temperature applications are not likely to become viable markets for the CCAR technology.

Table 6–9. Secondary Market Opportunities for CCAR Technology

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 21.

Economic Analysis

The economic analysis portion of the study provided sufficient information about the model, assumptions, and data to make it easy to follow and replicate. Two scenarios were evaluated: a conservative base case and alternative “optimal” scenario. The optimal scenario was said to be consistent with the market studies and input from food processing and refrigeration industry experts, making it clear that the base case is conservative.

The study set a time period of 2002–2016 over which to forecast likely economic benefits. Like the RTI and Ehlen case studies, this case study separately identified benefits estimated to accrue directly to the joint venture partners and those estimated to accrue more broadly. Also like those cases, this case study applied a counterfactual analysis in deciding how to attribute estimated benefits from the CCAR technology to ATP.

Table 6–10 shows a summary of projected base case cash flows for application of the CCAR technology in the primary market, food processing. The contribution of each of the four different types of benefits within this market area can be seen.

Table 6–11 shows three estimated measures of public returns from ATP’s investment in CCAR development: net present value (NPV), internal rate of return (IRR), and benefit cost ratio. Discounted at a 7% rate, the NPV was estimated at $459 million. The social return on total investment was not estimated. Because the study concluded that the technology would not have been developed without ATP assistance, the estimated benefits used to calculate public returns are presumably the same as would be used in calculating social benefits, but the costs presumably would differ.

Table 6–10. Base Case Cash Flows from Improved Quality, Yield, and Production Rates and from Reduced Refrigeration Costs from Application of the CCAR Technology for Food Processing

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 31.

Table 6–11. Base Case Net Present Value, Internal Rate of Return, and Benefit Cost Ratio (Calculated from the Cash Flows in Table 6–10) for the CCAR Technology

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 32.

Projected revenues accruing to the principal commercializing company in the joint venture were also presented. Discounted at a 9% rate, a rate selected by the researcher as a “likely proxy for the cost of funds of a major U.S. corporation,” the present value of these projected revenues was $64.8 million. According to the researcher, profits could not be estimated due to the required information being proprietary. Nevertheless, the information provided was sufficient to conclude that the public return is much greater than the private return.

Qualitative Benefits

At the time of the study, which was completed in early 2001, Pelsoci did not consider all the identified benefits of using CCAR technology to be quantifiable within the scope and budget of the study. He identified and discussed six additional categories of benefits, not included in the economic measures, but listed in Table 6–12.

Table 6–12. Additional Qualitative Benefits from Using the CCAR Technology

Source: Compiled from Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002.

It is typical that researchers encounter difficult- or impossible-to-measure effects when conducting economic case studies. But this does not mean the effects are unimportant. The approach used by Pelsoci in the illustrative case identifies and describes difficult-to-quantify effects qualitatively rather than ignoring them. The qualitative treatment reminds the reader of the effects not captured by the quantitative economic measures. In some cases, the study sponsor may wish to add resources to attempt further quantification. For example, techniques from health economics might be used to quantify the value of increased food safety.

Project and Portfolio Assessment Using Multiple Cases Studies with Uniform Collection of Key Indicator Data

While realizing the impracticality of performing in-depth case studies of every project, NIST and ATP management wished to harness the power of the case study for more projects and in a more systematic way. They wanted to develop a practical, cost-effective approach that would achieve 100% portfolio coverage to avoid selection bias, present each project story, consistently provide performance measures pertaining to the various dimensions of ATP’s mission, and begin implementation immediately.

The result was a new evaluative product, known as Status Reports. Status Reports feature short descriptive narratives, combined with the consistent compilation of key output and outcome data. Each completed ATP project has been the subject of a status report several years after its completion. By aggregating the uniformly collected data, and by using the data in the ATP’s composite performance rating system to score project overall performance (described in Chapter 8), ATP extended application of the case study method in ways to make it a more powerful tool for managing projects and a complex program, and answering a multitude of stakeholder questions. The second group of projects listed in Table 6–1 is the subject of this section.

To write the case studies of completed projects, analysts accessed ATP project records; used Business Reporting System (BRS) data when available; conducted telephone interviews with company representatives; conducted interviews with ATP project managers; searched company websites; used data collected by the U.S. Patent and Trademark Office; searched academic, trade, and business literature; searched news reports; viewed filings at the Securities and Exchange Commission; and used business research services, such as Dun and Bradstreet, Hoover’s Online Company and Industry Network, and CorpTech. They also took into consideration previously prepared in-depth project studies featuring economic analysis. The project’s lead company and ATP’s staff are asked to review each of the individual project write-ups for accuracy.

ATP’s First Published Collection of Status Reports

ATP’s first collection of status report was published in 1999 and was prepared by William F. Long, Business Performance Research Associates, Inc. ¹⁷² The report included case write-ups for the first 38 completed projects, a summary overview with aggregate statistics, and a brief treatment of terminated projects, that is, projects ended prior to completion. Featured in each case study, in an easy-tolocate text box, as illustrated below for one project is a summary of the key information uniformly compiled for all the projects.

Source: Long, Performance of Completed Projects, Status Report 1, 1999, p. 61.

In addition to the informational categories shown in the box, the narrative account of each project included an account of the role played by ATP. The report’s overview provided aggregate statistics on the characteristics of the 38 projects, the gains in technical knowledge, dissemination of new knowledge, and progress in commercializing the new technologies. For example, of the first 38, 15 had patents granted and 23 did not; 42% had published or presented papers and 58% had none. Figure 6–5, drawn from Long’s report, shows how employment changed at 27 small companies proposing to ATP as single applicants. Because these were for the most part extremely small, startup companies, dramatic increases in employment may signal project success and further progress.

Figure 6–5. Distribution of 27 Completed Projects at Small, Single Applicant Companies by Percentage of Employment Change

Source: Long, Performance of Completed Projects, Status Report 1, 1999, p. 14.

Table 6–13 shows summary results of ATP’s role. Two-thirds of the companies responding said they would not have proceeded without ATP funding; the rest said they would have proceeded but with a delay ranging from 18 to 60 months.

Taking the combined costs of the 38 completed projects and 12 projects that terminated during the same period, Long asked, “For its investment of $74.0 million, what has the public received, or is likely to receive, in return?” ¹⁷³

Indicating that it was beyond the scope of his study to estimate returns for the entire portfolio of 38 projects, Long turned for answers to the three projects in the group for which “detailed estimates have been calculated by other researchers.” These three included two of the projects from RTI’s set of tissue engineering studies—Aastrom Biosciences’ Stem-Cell Therapy Cost Reductions and Tissue Engineering’s New Materials to Repair Damaged Ligaments and the project studied by CONSAD Research—the Auto Body Consortium’s project on dimensional control for higher quality car bodies. Long concluded:

The value of the projected benefits resulting from the ATP contribution in just the three ATP projects ...would greatly exceed total ATP costs to date. ...Based on the investigations of projects conducted for this study, considerable evidence suggests that others among the 38 projects are also quite promising in terms of their future benefits potential. (p.18)

Table 6–13. ATP’s Role

Source: Long, Performance of Completed Projects, Status Report 1, 1999, p. 15.

ATP’s Second Published Collection of “Status Reports”

ATP recently extended its published collection of status reports to include 50 completed projects, an updated overview that provided a new project rating system built on the case study data, the addition of patent trees, and a more extensive treatment of terminated projects. ¹⁷⁴

Table 6–14 summarizes the output and outcome data collected for the status reports. The informational categories were selected to measure project progress toward achievement of the major ATP goals: (1) adding to the nation’s science and technical knowledge base—hence, information on awards by outside organizations for technical achievements, publications, presentations, and patents filed and granted; (2) disseminating the knowledge to others—a goal also furthered by publications, presentations, and patents, as well as by distribution (and reverse engineering) of commercialized products and processes, collaborations, and publicity value of awards; and (3) commercializing the technology in new and improved products and processes—signaled by attraction of capital for commercial activities, employment growth, commercialized products and processes on the market or expected soon; and future prospects. The informational categories included both outputs (e.g., publications and patents) and outcomes (e.g., commercial products).

The diverse output and outcome data collected for the completed projects are interesting and informative in their disaggregated form. They are more informative when analyzed statistically in an aggregated form, such as X percent of projects had resulted in commercialized products or processes two years after project completion. Nevertheless, it is difficult to gain a sense of how projects are performing overall when looking at nine sets of data linked to three aspects of mission. To provide a clearer assessment of performance on an experimental basis, the second volume of status reports featured a new rating system that scores projects based on a weighted composite of the nine types of data listed in Table 6–14. Called the Composite Performance Rating System (CPRS), it is a system that assigns 0 to 4 stars to each project on the basis of the composite of the uniformly compiled output and outcome data in the table. ¹⁷⁵ The CPRS is presented in more detail in Chapter 8.

Table 6–14. Status Report Data

Additional Status Reports in Preparation

The coverage of the next volume of Status Reports, according to ATP staff, is targeted at 100 completed projects. ¹⁷⁶ As the number of projects covered has grown, and the number of analysts performing the studies has also grown, the need for a set of data collection worksheets, and more rigor in assuring continued consistency in data collection, is apparent. Data templates are being used to guide preparation of the next batch of completed cases. In addition, a database has been developed that contains records for the first 50 projects, to which can be added data for future cases.

These advances in the use of case study for ATP—extending it to all completed projects, providing a common format, capturing data consistently that relate to achievement of ATP’s goals, aggregating the data across output and outcome categories, and using it to develop a composite rating—have given ATP a valuable new evaluation product built on case study, free of selection bias and useful for reporting portfolio performance.

Explicating Program Features and Exploring Program Dynamics

The following three studies, the last group in Table 6–1, illustrate the multiple and flexible uses of the case study method to explicate specific program features. They illustrate how case studies can be used in conjunction with survey, analytical, or empirical work. They also show how case studies can be stand alone reports on specific program elements not covered in larger studies, as in the Lide-Spivack account of the bottom-up processes that shaped ATP’s selection of a technological area for focused program competitions.

Financing Needs of High-Tech Startups

The case study portion of the Gompers-Lerner project was based on public documents and on interviews with seven Boston-area companies that received ATP awards, and covered the period from the firms’ establishment through fall 1997 when the case studies were completed. The companies span the industries of biotechnology, electronics, and software development. ATP’s intended purpose in funding the case studies was primarily to learn how and why the companies sought funding from ATP, and the role that the funding played. The authors also saw the seven cases as a source of information for formulating general recommendations for improving the program. ¹⁷⁷

Each of the seven case studies was divided into three sections. The first section provided a brief profile of the company’s technology, market focus, major milestones, and financial history. Special emphasis in this section was placed on factors that could affect how the company completed and later commercialized its ATP-funded research. The second section described the company’s ATP-sponsored project and examined the overall impact of ATP funding on the company. Among the topics recounted in this section were unanticipated research challenges, eventual project outcomes, the effect on the company’s research agenda, and the interplay between ATP grants and other public and private funding sources, although all of these topics were not covered in every case. The third section in the case histories described the company’s then current objectives (as of 1997), future plans, and recent developments.

Gompers and Lerner cited the case studies as evidence that ATP has had a substantial impact on the R&D activities of its awardees. They concluded that most of the company representatives interviewed felt strongly “that they could not have pursued their particular research challenges as quickly or as thoroughly without the ATP.” ¹⁷⁸

Combining State and Federal Programs to Advantage

Recognizing the mutually reinforcing role of state programs with its activities, in 1996, ATP entered into a non-financial memorandum of understanding with the Science and Technology Council of the States, designed to foster cooperation in outreach, technical and business assistance to applicants, and to facilitate the formation of joint ventures. Another interplay between ATP and the state programs occurs through university involvement in ATP projects. Between 1990 and September 2002, universities were research partners in 336 of the 642 projects funded by ATP. There are 166 different universities and 589 instances of their participation. Many state-supported universities provided critical research expertise and specialized laboratory facilities to companies that had won ATP awards. In turn, ATP funding indirectly augmented state funding for university-based research. To better understand how its activities meshed with those of the state programs, ATP commissioned a series of case studies of the firms that received support from both ATP and one or more state governments.

The four case studies were reported in volume 2 in Reinforcing Interactions between the Advanced Technology Program and State Technology Programs, by Feldman, Kelley, Chaff, and Fracas. The four subject companies were all technology- pioneering companies, defined as “new enterprises that from their inception were intent on developing and eventually commercializing new technologies.” ¹⁷⁹ A singular feature of such firms is that in addition to the typical set of problems associated with startup businesses, they have to invest in research and attract patient investment capital willing to support the development of a technology that may be years away from generating any revenues for the firm. This definition was employed to more precisely match the type of firm that was eligible for ATP support with the otherwise larger set of firms which would have been eligible for support under the wider latitude of eligibility criteria found across state programs.

Other differences between ATP and state programs were noted in the report. Central to these differences was that “as a program of the federal government, ATP is concerned with the development of technologies that benefit the nation as a whole.” Thus, an ATP project that led to the development of a new technology successfully commercialized by a U.S.-based firm and that generated spillover benefits would be considered a success, even though one or more firms participating in the original ATP project may not have directly benefited. “By contrast, to a state program intent on developing new businesses, success is largely measured in terms of the success of the individual firm and its growth within the region.” ¹⁸⁰

The study focused on the following questions: What state-provided assistance and programs do technology pioneers use, especially in the early stages? What kinds of linkages do technology pioneers have to other businesses and universities in the region? Are these linkages related to the resources needed by the company to carry out its own R&D, or are they important for bringing the technology to market and use by potential customers?

Selection of the four cases was based on recommendations of state technology program officers and the staff of ATP’s Economic Assessment Office. The four firms in the study—HT Medical Systems, SAGE Electrochromics, CuraGen, and AviGenics—were ATP award recipients between 1992 and 1998. The firms were located in four states—Maryland, New Jersey, Connecticut, and Georgia, respectively—but in fact, had drawn on program resources in a total of eight states. In each case, the study identified the stage of development of the company when it received assistance from a state government, the form of the assistance from state agencies and public universities, and how the assistance dovetailed with the federal government’s support of the R&D activities of these companies.

Conduct of the case studies required the cooperation of each company and the state agencies that had provided the assistance. Interviews with company principals, ATP program managers, officials at state development agencies, internal company documents, company prospectuses, and other documents in the public domain provided most of the information and data used to construct the case histories.

Although limited in number and not a random selection of all potential ATP cases, taken together the four case histories highlighted several important ways that ATP and state programs augment each other. Consistent with the underlying rationales of federal and state programs, the federal government played the largest direct role in funding R&D activities in all four cases, while state programs tended to support each firm’s R&D activities through university-based programs. State programs also tended to fund more downstream commercialization activities by enabling the companies to obtain access to specialized laboratory facilities and research capabilities of public universities, and by providing seed capital.

Explaining and Promoting a Program Area: ATP’s Information Infrastructure for Healthcare Focused Program

Bettijoyce Lide and Richard Spivack, both of ATP, used a case study approach to report on the specific history of ATP’s Information Infrastructure for Healthcare (IIH) Focused Program. The report provides insights to ATP’s external stakeholders, including both the executive and legislative branches, and to prospective applicants for ATP awards about ATP’s “bottom-up” decisionmaking processes.

Over the course of its history, ATP has employed a mix of competition mechanisms. From 1990 through 1998, it held General competitions open each year to all technologies. From 1994 through 1998, it awarded most of its funding through a series of 30 focused program competitions, “in which a suite of projects was funded to mobilize technology to address a particular problem.” ¹⁸¹

Starting with fiscal year 1999, ATP adopted a hybrid form of competition “in which ATP performs its outreach with industry much as it did under focused program competitions, but with a single competition open to all….” ¹⁸²

Lide and Spivack explained step-by-step how ATP’s IIH program was developed, from a call for white papers from the research community to the final scoping and approval of the focused program internally. They reported the technical and business goals of the program, and described the envisioned program as having a pyramid structure with infrastructural development technologies comprising the program’s base, user interface and efficiency enhancement technologies expected to be added next, and healthcare specific technologies to be developed last. In fact, projects of each kind were funded in each of the focused program’s competitions.

The authors used the program case study to reach out to the focused program’s community to allay concerns or uncertainties arising from an upcoming change in ATP’s competition structure to eliminate focused programs.

Summary of ATP’s Use of the Case Study Method

The studies reviewed in this chapter have illustrated the breadth and flexibility of the case study method as a means of evaluating the impacts of a project and of communicating a program’s features, activities, and impacts to multiple audiences. Case studies have helped ground ATP’s legislative mandates and program language in the operations of specific firms, industries, and research performers, thus making possible a clearer sense of the public interest served by the program. Case studies have highlighted in the context of specific firms the critical role that ATP funding has played in offsetting difficulties many firms had in obtaining venture capital or other capital to launch high-risk R&D projects. They have described the workings of joint ventures, as well as ATP’s bottom-up process for identifying focused program areas when ATP was using both focused and general competitions. They have detailed ways that ATP made operational key methodological and analytical concepts described in Chapters 2 and 4, such as counterfactual designs and social savings models, while focusing attention on challenges associated with converting non-market outcomes into measures amenable to impact analysis.

The reports covered in this chapter also have illustrated the limitations of the case study method. Some of these limitations, particularly the difficulty of generalizing from single cases, are characteristic of the case study methodology. Others relate to the confidential and proprietary nature of the data sought from the firms that were the ATP awardees. Still others relate to the specific heuristic use made of case studies in the early period of ATP’s evaluation program. Several of the case studies reported were designed to assess the feasibility of specific analytical techniques to estimate economic impacts; they were conducted early in the life cycle of ATP awards, typically prior to introduction of commercial products. Thus, most of the ATP economic case studies represent forecasts, not reports on realized outcomes.

The chapter also has introduced two approaches for increasing the ability to generalize from case studies to portfolio performance. One approach is to use a common method for in-depth analysis of similar technologies as RTI did in assessing the ATP-funded tissue engineering projects. A second approach is to use a common study template to collect key indicator data for all completed ATP projects and analyze the results statistically, as in the case of ATP’s Status Reports.

The chapter has shown the versatility, power, and limitations of the case study method. The case study method is a mainstay of evaluation because it tells an easily understood story complete with characters, goals, difficulties, and results. Moreover, it can tell the story in an interesting and memorable way, and provide extensive detail that may be useful in formulating theories and hypotheses and laying the groundwork for further evaluation, including economic analysis. Case study is a particularly useful evaluation method for making complex scientific research and technology development projects accessible to a wide audience.

____________________
148 In recent years, ATP has commissioned additional sets of in-depth case studies for certain technology areas—similar to the tissue engineering set presented in the next section below. These studies are underway at the time of this report and include case studies of a set of photonics projects, carried out by Todd Watkins, Lehigh University, and a set of component software projects carried out by Research Triangle Institute.

149 Albert N. Link, Advanced Technology Program: Economic Study of the Printed Wiring Board Joint Venture after Two Years (Gaithersburg, MD: National Institute of Standards and Technology, 1993); Albert N. Link, Economic Study of the Joint Venture Project on Short-Wavelength Sources for Optical Recording after Three Years of a Five- Year Research Program (Gaithersburg, MD: National Institute of Standards and Technology, 1994); and Albert N. Link, “Low-Cost Flat Panel Display Joint Venture [after Three Years of a Five-Year Research Program],” in Evaluating Public Sector Research and Development (Westport, CT: Praeger, 1996).

150 Link, Advanced Technology Program : Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997.

151 In the case of the advanced display project, Link found that the joint venture members directly competed with one another, and that this pattern of competiton reduced their willingness to share research results.

152 These were omitted from the calculation because there was no base from which to compute savings.

153 Sheila A. Martin, Daniel L. Winfield, Anne E. Kenyon, John R. Farris, Mohan V. Baal, and Tayler H. Bingham, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, GCR 97–737 (Gaithersburg, MD: National Institute of Standards and Technology, 1998). 154 Ibid., pp. 1–2.

156 See George W. Torrance and David Feeny, “Utilities and Quality-Adjusted Life Years,” International Journal of Technology Assessment in Health Care, 5: 559—575, 1989.

157 For an account of the QALY technique in estimating net benefits of new medical technologies, see Andrew Wang, “Key Concepts in Evaluating Outcomes of ATP Funding of Medical Technologies,” The Journal of Technology Transfer 23(2): 61–65, 1998.

158 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–22. (Note that the report uses the page numbering system of chapter-page.)

159 Diabetes Control and Complications Trial Research Group, “The Effect of Intensive Treatment of Diabetes on the Development and Progression of Long-Term Complications in Insulin-Dependent Diabetes Mellitus,” New England Journal of Medicine 18:1468– 1478, 1996.

160 Josephine A. Mauskopf and Michael T. French, “Estimating the Value of Avoiding Morbidity and Mortality from Foodborne Illnesses,” Risk Analysis 11(4):619–631, 1991. 161 Michael J. Moore and W. Kip Viscusi, “Doubling the Estimated Value of Life: Results Using New Occupational Fatality Data,” Journal of Policy Analysis and Management 7(3):476–490, 1988.

162 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–2.

163 CONSAD Research Corporation, Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing, NIST GCR 97–709 (Gaithersburg, MD: National Institute of Standards and Technology, 1996).

164 Mark A. Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, NISTIR 6373 (Gaithersburg, MD: National Institute of Standards and Technology, 1999).

165 CONSAD Research Corporation, Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing, 1996, p. 10.

166 Ibid., p. 17. 167 Ibid., p. 18.

168 Thomas Pelsoci, Closed-Cycle Air Refrigeration Technology for Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, NIST GCR 01–819 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

169 Ibid., p. v.

170 For example, if there is a 50% probability of achieving technical success, and a 50% probability of achieving commercial success, the combined probability of success is 25% (i.e., 0.50 multiplied by 0.50).

171 Ibid., p. 6.

172 William F. Long, Performance of Completed Projects, Status Report 1, NIST Special Publication 950–1 (Gaithersburg, MD: National Institute of Standards and Technology, 1999).

173 Ibid., p. 16.

174 Advanced Technology Program, Performance of 50 Completed Projects, Status Report 2, NIST Special Publication 950–2 (Gaithersburg, MD: National Institute of Standards and Technology, 2001).

175 For a fuller treatment of the CPRS, see R. Ruegg, A Composite Performance Rating System for ATP-Funded Completed Projects , NIST GCR 03–851 (Gaithersburg, MD: National Institute of Standards and Technology, 2003).

176 KPMG Corp., (now BearingPoint) is preparing the next group of studies in the Status Report series. Information on the coverage of the next release was provided by Stephanie Shipp, Director, Economic Assessment Office, ATP.

177 More often, the business case study method used by Gompers and Lerner is seen as a source of information to formulate recommendations for improving the subject companies rather than the government program that provided financial assistance.

178 Ibid., p. 39. 179 Feldman et al., Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, 2000, p. iii.

180 Ibid., p. 3.

181 Bettijoyce Lide and Richard N. Spivack, Advanced Technology Program Information Infrastructure for Healthcare Focused Program: A Brief History, NISTIR 6477 (Gaithersburg, MD: National Institute of Standards and Technology, 2000), p. 1.

182 Ibid., p. 1.

Return to Table of Contents or go to Chapter 7: Econometric/Statistical Method.

Date created: July 22, 2004
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)