A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 6:  Case Study Method

ATP has used case studies for multiple purposes throughout its first decade. Case studies have helped make the technical and economic aspects of its complex technology projects more accessible to non-scientists. Case studies have helped explore the genesis of projects and programs, and tell the stories of the people and organizations behind the projects. Case studies have helped answer why and how questions, explain roles and goals, investigate project dynamics, track progress, identify market applications, measure outcomes, and—performed in multiples—estimate portfolio performance.

All of the case studies presented in this chapter describe subject projects, organizations, and technologies. Reflecting ATP’s emphasis on demonstrating economic impact, many of them include substantial quantitative elements, including projections of social rates of return. Table 6–1 lists a selection of ATP’s case studies on which this chapter is based. The main objectives and key features of the studies are shown.

This chapter is organized according to three major case-study objectives: modeling and estimating economic impacts, estimating project and portfolio performance using multiple cases and progress indicators, and explicating selected program features and exploring dynamics. The chapter emphasizes the approaches and models used to carry out the case studies; findings are presented only as they contribute to a fuller explanation of the underlying models or cases.

Economic Case Study of Individual Projects

The case studies illustrate the point made earlier that evaluation is an art as well as a science and a craft. Each researcher or team of researchers takes a somewhat different approach to creating a case study, depending on project particulars—objective timing of the study, market applications, data availability—as well as their research expertise, study budget, and research perspective. The first group of case studies provides economic impact  stimates for individual projects funded by ATP. These cases also provide descriptions of the technologies and the innovating organizations, the sources  of economic benefit, and the role of ATP in the projects. The first approach reported here was developed for application early both in ATP’s history and in the life cycle of the surveyed projects. Lacking market observations on economic benefits at the time the studies were undertaken, the researcher took
a novel approach to estimating minimum project benefits based on research cost savings. The second approach covered seven tissue-engineering projects, all aimed at providing improved medical treatment at lower cost. The team of researchers developed a method for estimating expected social economic return on public investment, using concepts from health care assessment to measure patient quality-of-life benefits in several of the cases. The third approach listed combined microeconomic estimation techniques with macroeconomic modeling to estimate national benefits from adoption of new automotive technology. The fourth approach focused greater attention on market research as a way to explore multiple target applications for the technology, emphasized combining estimated effects from multiple benefit streams, presented an explicit treatment of qualitative benefits in conjunction with quantitative estimates, and provided a more transparent exposition.

Table 6–1. Ten of Sixteen Studies Featuring Case Study Represented*

* Note: Other studies using the case study method as a secondary method listed in Table 3–4, but not explicitly treated here, are by Austin and Macauley, 2000; Przbylinski, 2000 draft; Fogarty et al., 2000 draft; Liebeskind, 2000 draft; and Dyer and Powell, 2002.

Each of these studies is unique, raising the standard evaluation question of how to generalize findings from case studies. Adding weight to this reservation is that, aside from the set of tissue engineering studies, each was a separate undertaking commissioned at a different time, and no attempt was made to establish a uniform set of questions, data collection procedures, or other uniform research protocols among the studies by different researchers. This decentralized approach was considered appropriate at the time the studies were commissioned as ATP was purposefully experimenting with different approaches and testing the analytical capabilities of contractors. Each of these studies was directed at a somewhat different question and in a different set of circumstances. Each of the studies represents a legitimate approach to the particular case and set of problems it tackled; in the aggregate, they highlight the flexibility as well as strengths and weaknesses of case studies within a larger portfolio of evaluation techniques. ¹⁴⁸ The key features of each of this first group of studies are summarized in turn.

Estimating Minimum Benefits of New Technology in Terms of Research Cost Savings and Competitive Improvements

In its first competition in 1990, ATP funded five joint ventures among a total of 11 projects selected—several of them relatively large, five-year efforts. Eager to learn more about how these early joint ventures were performing and to document results, ATP commissioned Albert Link, University of North Carolina- Greensboro, to evaluate three joint ventures near the midpoints of their five-year duration. ¹⁴⁹ Each of the three projects represented an attempt by a group of U.S. companies within an industry sector—the printed wiring board (PWB) industry, the data storage industry, and the advanced display industry—to respond to foreign competition and enhance their industry’s competitiveness through the development of a suite of “leap-frog” technologies. Subsequently, ATP focused on evaluation of the PWB project, and sponsored Link to update the analysis of the joint venture at project end. ¹⁵⁰

Link’s Approach

Link used essentially the same approach in each of the case studies he performed: He described the industry, the technology, the nature of the collaboration, the major research tasks, the project’s organizational structure, and the role of ATP. He also identified changes in the participating organizations and research plan as the project unfolded. Because of the early stage of the projects he investigated, Link focused on quantifying research cost savings from the collaboration and on changes in competitiveness, rather than on attempting to forecast benefits from the technology in use. In each case, he surveyed the participants to collect data needed to estimate impacts.

By examining the characteristics of the member companies, Link assessed the nature of the collaboration. For example, although the PWB joint venture is primarily a horizontal collaborative research arrangement, Link found that the members were actually not head-to-head competitors. This finding was important because it helped explain why the joint venture members collaborated more fully and shared their research results more extensively than in cases where joint venture members were direct competitors. ¹⁵¹

Data Collection

To collect data for the quantitative part of the case study, Link used a survey with three parts, and one with a counterfactual element. The survey part of the PWB case study was described in earlier in Chapter 5.

Link collected additional data by asking members of the project’s steering committee—a management group made up of representatives from the participating organizations—to respond, in terms of the level of agreement/disagreement, to a set of ten statements. The statements describe the importance of the PWB joint venture to the company and the industry in terms of ability to refine manufacturing technologies and commercialize new scientific discoveries and technologies more rapidly and to improve competitive position.

Link collected qualitative information from members of the steering committee, who were asked to complete the following statement: “My company has benefited from its involvement in the PWB joint venture in such non-technical ways as ...”. Also, they were asked to listen to a reading of the goals of ATP, and to indicate in response the degree to which they thought ATP goals had been fulfilled in their project.

The remaining data Link collected in support of the impact analysis related to effects from using the resulting technology. He asked members of the steering committee to estimate their own company’s productivity gains traceable to using project outputs in their production. These data were sparse because the research was just concluding at the time of the study.

Presentation and Interpretation of Results

Link used the collected data to provide estimates of minimum impact for the PWB joint venture, part of which are quantitative and part qualitative. To obtain an estimated minimum dollar value of the assistance provided by ATP, Link combined the various cost savings from efficiency gains in carrying out the projects as a joint venture. He counted costs savings only for that part of the research the companies said they would have pursued on their own without ATP, because otherwise they presumably would have incurred no research costs.

Table 6–2 summarizes direct impacts to member companies: research cost savings (a total of $35.5 million at project end), production cost savings ($5.0 million at project end), and indirect impacts on member companies (that is, increase in competitive position in world markets). It also shows partial spillovers to the PWB industry: 214 papers, 96 conferences, and increased competitive position for the U.S. industry as a whole. For comparison, the table brings forward the summary results of the earlier case study of the PWB project that Link performed two years into its five-year timeframe.

In addition to the results summarized in the table, Link pointed out potential value in the new capabilities the companies now have due to approximately half of the total of 62 project research tasks that were omitted from the cost savings calculation. ¹⁵² He also noted reduced cycle times for new project and process development and the presence of substantial technology transfer products providing pathways for the rest of the industry to benefit from the projects outputs. As is often the practice in case studies, Link included representative anecdotal responses from the companies about how they have benefited.

Link presented the results as “a conservative lower-bound estimate of the longrun economic benefits,” and as “partial and preliminary estimates of project impacts.” He pointed out that the bulk of production cost savings and performance gains would be realized in the future as the technology results diffuse and are more widely implemented.

Table 6–2. Summary of Survey Findings on Partial Early-Stage Economic Impacts

* These impacts are based only on those research tasks that the members thought they would eventually have done without ATP, and not the cost and time savings associated with the new capabilities resulting from those tasks that they would not have done at all without ATP.

Source: Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. 34.

Modeling Private and Social Benefits of a Set of Related Medical Technologies

Among the more ambitious and methodologically important case studies commissioned by ATP over its first decade was that conducted by economists at Research Triangle Institute (RTI) to estimate the economic impacts of a portfolio of seven ATP-funded projects in medical technology. ¹⁵³ The study is valuable for several reasons. As an approach to portfolio assessment, it illustrated how use of a common, consistent methodology across a set of technologies within the same industry can be used to identify “project” or “technology” characteristics that affect the relative economic impacts of these projects. The study also showed how a formal model of relationships can be used to guide collection of information and data, and is notable for the care and detail with which it assessed ATP’s programmatic objectives in the context of specific technologies. Finally, the study has value because, among those presented, it most explicitly links its design to the central analytical models and concepts used to articulate ATP’s mission, while at the same time addressing “specific methodological challenges that have not been addressed in ATP’s previous methodological development efforts.” ¹⁵⁴

Study Objectives

The study had three objectives: First, to develop a methodology for estimating the expected social rate of return on public investment in ATP-funded projects with medical applications; second, to apply the model to all of the ATP-funded multiple-application tissue engineering projects funded by ATP between 1990 and 1996; and third, to estimate the composite social return and compare it with the composite private return for the set of cases. As shown in Table 6–3, four cases were performed in greater depth than the others.

Table 6–3. Overview of ATP Projects Included in this Study

Note: Tissue engineering produces materials that can be used either to replace or correct poorly functioning components in humans or animals. Throughout this report we refer to each project by the abbreviated title listed below the full title.

* BioHybrid was approved for a 2-year no cost project extension.

Source: Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–13.

RTI’s Approach to Estimating Benefits and Costs

RTI’s approach was to build on Mansfield’s model for estimating private and social rates of return, modifying it to take into account the specific forms of benefits generated by medical technologies. It also incorporated the evaluation and policy design precept implicit in Mansfield’s work and made explicit by Jaffe: that because private sector R&D tends to generate social rates of return, the test of ATP’s economic impacts are the social rates of return it generates above those likely to have resulted from private sector activities alone.

RTI modeled ATP funding of R&D projects as affecting the development of medical technology in three ways: (1) accelerating the technology’s benefits (i.e., bringing benefits to the private sector, patients, and society sooner and for a greater number of years than without ATP funding); (2) increasing the likelihood of success (i.e., increasing the amount of R&D conducted and thereby the likelihood that a project will be technically successful); and (3) widening the scope of the project and enabling the company to apply its technology to additional diseases or patient populations. Figure 6–1 illustrates the model underlying the selection of relationships and variables for which information and data were collected.

Figure 6–1. Elements Determining Social Return on Public Investment and Social Return on Investment

Estimation of each of the generic effects, in turn, represented construction of a set of scenarios detailing what was expected to happen because of ATP funding relative to what would have happened in the absence of the funding. Thus, for example, if it were assumed that ATP funding accelerated bringing a product to market by two years, the model assumes that the with-ATP innovation starts generating benefits two years earlier and has an economic life two years longer (and thus a higher net present value) than the same innovation produced without ATP funding.

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–5.

Table 6–4 relates these differential benefits across tissue engineering projects to the effects of ATP funding. The single greatest source of differential effects was estimated to be acceleration by ATP of the rate at which a technology is brought to a marketable stage. Company officials involved in developing biopolymers for tissue repair, in RTI’s words, reported that without ATP assistance the company might not have developed this technology at all or might have developed it so slowly that the market opportunity for it would have passed before it was ready for commercialization. In this case, the study assigned a 10-year advantage in estimating project benefits with ATP support.

Table 6–4. Impact of ATP Funding on the Development of  Medical Technologies for Seven Tissue Engineering Projects

Note: Our model allows conceptually for ATP funding to widen the scope of a project. In practice, for the applications in this study, there was little or no impact in all but two cases, which we did not quantify. *This is the number of years of acceleration reported by the ATP-funded companies. For the one to two year ranges, we used the lower number for our analysis. For the three to five year range, we used the midpoint of the range.

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–23.

RTI modified the Bass diffusion model ¹⁵⁵ to estimate adoption of the new technologies. The rate of adoption was increased during the earlier period and decreased as the market potential was approached. RTI assumed that a newer ¹⁵⁵ Frank M. Bass, “A New Product Growth Model for Consumer Durables,” Management Sciences , 15(5):215—227, 1969. technology would completely supersede each of the ATP-funded technologies after a 10-year period and models a cessation of diffusion at that time.

RTI separated net benefits estimation into those occurring in the medical technology sector and in the health care delivery sector. For the medical technology sector, net benefits included estimated change in revenues from sales of the new medical products and procedures, less investment and production costs incurred in bringing them to market, as compared with the displaced defender products and procedures; that is, the change in profits from having the new technologies. For the health care delivery sector, net benefits included reductions in the costs of health care and the value of increased health benefits to patients.

To estimate the value of health benefits, RTI adopted a concept called Quality Adjusted Life Year (QALY), developed in the field of healthcare to allow quantification of health changes in terms of the quantity and quality of life. ¹⁵⁶

Where a year of life at full health is assigned a QALY value of 1.0, and death is assigned a value of 0.0, in between states are assigned QALY values between 0.0 and 1.0. ¹⁵⁷ The QALYs must be translated into dollar values. The steps required in valuing per-patient changes in health outcomes and RTI’s methodological approach at each step are summarized in Figure 6–2.

Figure 6–2. Valuing Per-Patient Changes in Health Outcomes

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–9.

As noted, one of the contributions of the RTI methodology is that it offered insights into the sources of variation in rates of return across a portfolio of similarly directed technologies. As stated in the report, “Social returns to these projects can vary with respect to the number of patients treated, the value of the health benefits of the new technology, their impact on health care costs, and the probability of technical success.” ¹⁵⁸

Data and Assumptions

Information on market potential, R&D expenditures, benefits to patients, and other variables necessary to compute social and private rates of return for each case was collected from a number of sources, including representatives from the companies receiving ATP funding. According to Martin et al.:

The most important sources of information about each technology were representatives of the companies receiving ATP funding. We interviewed representatives of each lead company and, in some cases, also interviewed representatives of partner companies. We also talked with a number of physicians and consulted a variety of secondary data sources, including medical literature and statistical databases, to develop estimates of costs and benefits. (p. 1–14)

The inclusion of estimated benefits from health improvements was dependent on the researchers being able to find existing QALY values, because estimating them was beyond the scope of the project. These values have been developed for certain health conditions and diseases from surveys of affected populations, such as cancer patients and diabetics, based on choices expressed by respondents; however, they are not available for every disease or condition. Where they found suitable QALY data, the researchers used the data to develop benefits from improved health outcomes. For example, the researchers found much of the data required for the model of health outcomes related to new treatments for diabetes from the Diabetes Control and Complication Trial (DCCT). ¹⁵⁹ They found, for example, that blindness from retinopathy carries a QALY of 0.69; end-stage renal disease carries a QALY of 0.61; and lower extremity amputation, a QALY of 0.80. For illustration, Table 6–5 lists various QALYs for different health states and corresponding study source.

Table 6–5. Comparison of QALY Utility-Weights for Different Health States

* For biopolymers, the two sets of figures are identical because all of the social return can be attributed to ATP investment.
**See notes to Table 6.5 in the original for an explanation of the derivation of the composite measure of return.

Source: Excerpted from RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 2–23.

To determine the dollar value of the change in the patient’s well being, RTI researchers estimated the economic value of a QALY based on willingness-to-pay values for avoiding illness and accidents taken from existing studies. ^{160, 161} They also drew probability data from existing studies, such as the probability of blindness given diabetes from the DCCT study.

Composite Private and Social Rates of Return

Table 6–6 summarizes the study’s estimated expected social return on total investment and the expected social rate of return on public (ATP) investment for each of the ATP projects examined in the RTI study. It also shows the composite rate for all the projects taken together.

Table 6–6. Social Return on Investment and Social Return on Public Investment: ATP Projects in Tissue Engineering for a Single Preliminary Application

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies , 1998, p. 1–22.

Based on these results, the authors concluded:

ATP funding is responsible for inducing about 31% of the total social returns from all of these projects over 20 years. For the individual projects, the effect of ATP on social returns ranges from about 25% to 100% of the social returns. (p. 1–22)

Table 6–7 reports the composite private return on investment for the seven projects. Based on comparing the social and public returns in Figure 6–7 and the private returns in Figure 6–8, the authors concluded:

The wide disparity between social and private returns indicates the importance of ATP incentives to the private sector to pursue these technologies. Because the social returns far outweigh the returns to the companies developing, commercializing, and producing these high-risk projects, the private sector may under invest in these kinds of high-risk projects. (p. 1–24)

Table 6–7. Composite Private Returns: ATP Projects in Tissue Engineering for a Single Preliminary Application

Source: RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, p. 1–24.

Limitations

As careful, systematic, and methodologically focused as the RTI study was, it still has limitations. Paramount among these, as with many of the case studies reviewed in this section, is that it is a projection of expected net economic benefits, not a measurement of observed benefits. None of the tissue engineering technologies covered in the RTI study had entered commercial use at the time of the study, although some were in clinical trials. In fact, at the time of the study, it had not yet been demonstrated fully that all would function technically as expected, thereby compounding uncertainty in the estimated outcome. Thus, there was a “shortage of ex post empirical data.” ¹⁶² This limitation, clearly, was a function of the time at which the case study was done, and not a function of the case study method or the implementation of this case.

Another limitation of data rather than the model is the fact that the study only estimated patient benefits from improved health outcomes when there were preexisting QALY data for the relevant medical conditions. Thus, the disparity in the size of net benefit estimates among the projects to some extent reflected the inclusion of patient health care cost, but not of patient health outcomes in several cases. Exercising the model for only one application of multi-use technologies is a choice reflective of budget limitations rather than a shortcoming of the model.

What most distinguishes this study is its explicit attention to methodological development; linkage to the multiple, attributed impacts of ATP funding, and formal ties to core theoretical constructs. Further, by analyzing all the projects funded by ATP within a single technological area, the study strengthened its ability to generalize results within that area.

Estimating Market-Based Economic Impacts from Automotive Technology Combining Microeconomic and Macroeconomic Modeling

This section pairs two case studies: CONSAD Research Corporation’s case study of dimensional control technology, the “2mm Project,” ¹⁶³ and former economist at the National Institute of Standards and Technology (NIST) Mark Ehlen’s case study of flow-control machining technology. ¹⁶⁴ Although performed by different researchers, these two cases bear similarities. In both cases the first application of the multi-application technologies was to automobiles. Both entailed primarily vertically structured joint ventures, which bring together in some role supplier-innovators, universities, and large automobile assemblers. Both dealt with new manufacturing process technologies that offered quality/performance improvements. Both employed a micro-level examination of impacts arising from the technical characteristics of the project. Both attempted to link microeconomic modeling of firm- and industry-level impacts to macroeconomic modeling of national economic impacts.

The case studies have several important differences. The first case study was done on a much smaller budget, a shorter schedule, at an earlier time, on more of an experimental basis, and with less detail than the second. The first case study uses two, largely disconnected approaches: a microeconomic approach to estimate production and maintenance cost savings based on unit savings and current production volumes, and a macroeconomic approach to estimate total industrial output and employment changes, based on expert judgment about the increase in sales of U.S.-made vehicles due to technology-based quality improvements. In contrast, the second case study systematically built its model from firm level, to industry level, to the national level, integrating across the micro- and macro-parts of the analysis. Features of these two cases are discussed and compared below.

Establishing Impact Expectations by Examining Technical Characteristics

Both of the case studies explained to the reader why and how the projects’ technical accomplishments could logically be expected to yield benefits, an important part of building the case story. In the case of the 2mm technology, CONSAD researchers explained that U.S. auto assembly plants required a cost-effective method of reducing dimensional variation in auto body assembly, using the existing workforce. The project developed a new metrology-based process for improving the fit of discrete manufactured parts, with potential application to multiple manufacturing industries. Four types of direct benefits were expected from its application to automobile manufacturing: (1) decreased production costs, (2) decreased product maintenance costs, (3) improved product quality, and (4) reduced time required to launch new products or product models. Experimental implementation of the technology in five U.S. auto assembly plants at the time of the study provided CONSAD with estimates of unit cost reductions.

In the flow-control machining project, Ehlen explained how the multi-application technology increases the functional precision of cast-metal parts that carry fluids in interior passageways. Applied to auto engines, the improved precision can increase engine horsepower, increase fuel efficiency, reduce emissions, and reduce engine costs. Ehlen provided diagrams showing how the efficiency of combustion is improved. As in the previous case, performance was informed by actual data—in this case, from testing a prototype-working machine on engine manifolds. Ehlen related how the improved technical capabilities could potentially be deployed in the auto industry in alternative ways, affecting the resulting benefits. For example, in the face of fuel shortage, it could be applied in producing engines for all vehicles to decrease fuel consumption across the board. It could be used to meet increased Corporate Average Fuel Economy (CAFE) requirements. It could be used in some vehicle lines in order to sell other, less fuel efficient models, while still meeting overall the existing CAFE requirements. It could be used in specialty vehicles to increase horsepower. In other words, there are a variety of possible strategies for deploying the new technology.

Investigating the Role of ATP

Both of the case studies addressed the role of ATP as they considered why federal assistance was needed, particularly given that large end-user companies were present in both joint ventures. They came to much the same multi-reason conclusion about why the project would not likely have gone forward without ATP involvement. As Ehlen writes:

Ford and GM are unlikely to unilaterally adopt a new process that has not been proven to work; the FCM [flow-control machining] processes are particularly challenging since both constitute a radical departure in finishing processes—manufacturing directly to functional performance.  Ford and GM are also unlikely to directly collaborate on a new process, since they are direct competitors on routine business matters and have concerns about federal antitrust-law enforcement. They tend not to fund the research of their suppliers. The suppliers would not perform the research themselves; they generally do not have the capital to do extensive in-house research—particularly not high-risk research. University researchers are typically interested in doing their own research, not the research of a supplier to automakers, and are not able to self-fund the type of research. (p. 4)

CONSAD researchers emphasized the difficulties of achieving cooperation among industrial participants who frequently compete against one another, or forging a joint research undertaking among different members who “might realistically expect notably different returns from their involvement in the project.” ¹⁶⁵ In the judgment of the authors:

It appears unlikely that (a) this complex joint venture could have been formed and (b) funding for the research project could have been coordinated without direct administrative and financial involvement by the federal government. (p. 10)

Modeling the Technology’s Adoption by Auto Manufacturers

The two case studies differed considerably in their modeling of the take-up of each technology. Based on reported steady adoption of the 2mm technology by a growing number of assembly plants at the time of the study, CONSAD researchers assumed successful commercialization within the automobile manufacturing industry within a relatively short period of time.

In contrast, Ehlen included as a major part of his study the assessment of the likelihood that the automobile industry would implement the flow-control machining processes, outlining two implementation paths for estimating nearterm and longer-term impacts. He used historical adoption models of similar fuel-efficiency enhancement by auto manufacturers in modeling adoption of the new processes. Figure 6–3 illustrates the adoption modeling. The horizontal portion of the heavy solid line shows the near-term conservative view that the processes would be adopted at an introductory level only, maintained for five years, and then dropped. The upward sloping portion of the line indicates the longer-term, more optimistic projection of a broader implementation at the historical adoption rate of fuel injection technologies, implemented over 20 years by 80% of the market.

Figure 6–3. Historical Adoption Rates of Technologies Enhancing Fuel Efficiency, and Implementation Rates used by Ehlen in the Case Study

Source: Ehlen, Economic Impacts of Flow-Control Machining Technology: Early Applications in the Automobile Industry, 1999, p. 54.

Data and Assumptions

Both case studies were limited in their assessment by the recentness of the technological innovation, and the absence of market-based data. As stated by CONSAD researchers:

Because the technologies developed by the 2mm Project are new, their impacts on industrial production and economic activity are not yet revealed in the extant empirical data on industrial performance. (p. 17)

In the absence of market data, the CONSAD team turned to two expert panels for estimates of the magnitudes of impacts. The first group of experts was composed of individuals knowledgeable about the substance of the technologies and their likely impacts on costs and quality. The individuals interviewed were primarily university researchers, manufacturing engineers, and technicians and engineers involved with the initial implementation of project results at five automobile assembly plants. The second group was composed of individuals knowledgeable about the industries and markets in which the technologies would likely be used; these experts were asked for their assessments about the expected extent and rate of adoption of the technologies in specific industries and markets.

Limiting validation of the work of these two panels was the lack of detail provided due to concerns about confidentiality. “The individual sources of information and judgments, and information for individual plants and firms adopting technologies that have resulted from the 2mm Project are not cited because of the proprietary and confidential nature of the data about current and expected cost savings and expected product demands.” ¹⁶⁶ Similarly omitted in the study’s report was the means by which the judgments of the two panels were put together. “The plausibility of the judgments provided by the two groups of experts has then been evaluated by examining the coherence among the judgments provided by the various experts in each group.” ¹⁶⁷ Thus, the CONSAD study lacked transparency.

Ehlen seems to have faced fewer obstacles in obtaining and citing industry and firm data due to company confidentially. In general, the data, assumptions, and step-by-step procedure are more transparent in Ehlen’s study. Ehlen received close cooperation, particularly from the major innovator, Extrude Hone, who took a keen interest in the case study and seemed unusually willing to share data. CONSAD also received close cooperation from the companies in the joint venture it studied, but apparently faced more restrictions on the publication of data. Without cooperation from project participants and the ability to attribute data to sources, a researcher will have a difficult time conducting a detailed and replicable case study.

Using Macroeconomic Modeling in the Case Study

Both studies used macroeconomic modeling to estimate national impacts from using the technology in the auto industry. In fact, the major methodological fillip to these studies relative to other ATP case studies was the effort to scale up economic impacts through the use of a macroeconomic inter-industry model. They both used the REMI (Regional Economic Modeling, Inc.) model for this purpose.

The application of REMI in these two projects defined the limit of ATP’s use of macro-economic modeling as an adjunct to case study over its first decade of evaluation. Attempting to use macroeconomic modeling to assess the impact of a project, or even an entire program, is controversial. The “noise” in a $10 trillion economy is likely to overwhelm the measures of a macroeconomic model of the U.S. economy. Yet, the REMI model, comprised as it is of regional components and a set of structural equations linking inputs and outputs, prices, and consumer spending, offers the possibility of estimating project impact at the national level, provided the subject technology will have sufficient impact to show up at an industry-wide level and can be effectively captured in the model’s variables and causal linkages. In both the case studies treated here, it was thought that the extensive participation of large auto manufacturers provided conditions that would allow REMI modeling to be used. But for most ATP projects it is unlikely that necessary conditions would be met, and a macroeconomic model would not be an appropriate evaluation tool.

The CONSAD study applied the REMI Economic and Demographic Forecasting and Simulation 53-Sector (EDFS–53) model in conjunction with analysis based on the input-output (I-O) tables of the U.S. Department of Commerce’s Bureau of Economic Analysis. The model was used to estimate changes in industrial production and employment due to the projected increase in autos resulting from an increased combined market share of the participating U.S. auto manufacturers, based on expert opinion about the change in demand for U.S. assembled autos due to improved quality.

In contrast, Ehlen used a more detailed REMI model, and systematically built and integrated from the microeconomic modeling to the macroeconomic modeling. First, he estimated the impact on firms of near-term implementation over a fiveyear implementation path. Next, he estimated changes in industry performance and the change in annual sales for the three industry sectors involved in the supply of the technology. Finally, he used market quantities in the REMI analysis to estimate macroeconomic impacts. Table 6–8 summarizes the REMI findings for the year 2004, based on the assumed five-year, conservative implementation path.

Table 6–8. Annual Impact on U.S. Macroeconomy of Near-Term, Five-Year Implementation Path: Year 2004

Note:  Dollar concepts are in constant dollars.

Source: Ehlen, Economic Impacts of Flow-Control Machining Technology: Early Applications in the Automobile Industry, 1999, p. 46.

Estimating Net Benefits from Multiple Applications of an Advanced Refrigeration Technology

Whereas the case studies presented in the two preceding sections each performed a benefit-cost analysis for the single most promising application of the technology, the case study presented in this section investigated multiple applications. Prepared by Thomas Pelsoci, managing director of Delta Research Company, the case study examined closed-cycle air refrigeration technology (CCAR), funded by ATP in 1995. ¹⁶⁸ The joint venture project was completed in 1999, and, after subsequent corporate product development efforts, yielded “a cost-effective system for delivering ultra-cold refrigeration in the -70ºF to -150ºF temperature range to food processing, volatile organic compound, and liquid natural gas applications.” ¹⁶⁹ The system uses environmentally benign dry air as the working fluid to replace harmful refrigerants.

This study has several features that make it a good example of an economic case study. It has a clear technical characterization of the technology and its state of development; an assessment of the functional capability of the technology; an analysis of potential markets; description of pathways to commercializing in those markets; an assessment of market demand; a straight-forward, transparent benefit-cost analysis with clear identification of data and assumptions; discussion of the counterfactual; estimation of both private and social benefits; and inclusion of qualitative benefits.

Attention to Test and Demonstration Results

Given the prospective approach of the benefit-cost analysis, the attention the study gave to results of tests and demonstration of CCAR in operation takes on added importance. When technical feasibility, in addition to market feasibility, is in question—as it was in several of the tissue engineering case studies examined earlier—project risk is substantially increased. ¹⁷⁰ To address the question of CCAR’s technical feasibility, ¹⁷¹ Pelsoci cited the conclusion of project participants that “CCAR met or exceeded all acceptance criteria and successfully demonstrated its technical feasibility.” Thus, the technology has been demonstrated to work, freeing the researcher to focus on the question of whether it will be adopted, when, and for what uses.

Market Assessment

The market analysis emphasized fact-finding and analysis of both primary and secondary markets for the technology. CCAR was termed a niche technology because it represented a cost-effective alternative only within the specified temperature range. Mechanical refrigeration provides cooling above the -70ºF range. Cryogenic refrigeration provides cooling below -70ºF, but its high cost may limit industrial applications.

In the U.S. food industry value chain, the study identified “further-processed foods,” a $131 billion market, as the targeted primary end market for the CCAR technology. As illustrated in Figure 6–4, the study identified key market drivers of this market segment.

Figure 6–4. Market Drivers for the Further-Processed Food Industry

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 12.

The study related each of the market drivers to changing demographics. It explained how colder freezing is linked to more rapid freezing and in turn to higher quality, and how the CCAR technology provides an enabling technology for meeting market demands in the targeted primary market segment. It sourced two existing market studies by independent market research companies to assess the level of interest among food companies for the CCAR technology. It also relied on information gleaned from discussions with expert technical and sales staff at the joint venture companies, food industry associations, and food companies.

The study identified five promising pathways for marketing CCAR refrigeration services for food processing based on primary research and analysis completed during 2000 and early 2001: (1) replacing liquid nitrogen as a refrigerant, (2) replacing carbon dioxide as a refrigerant, (3) installing CCAR units at plants with expanding production, (4) installing CCAR units at newly constructed food plants, and (5) exporting into the overseas market. The study identified the four potential secondary markets for the CCAR technology shown in Table 6–9, and discussed the opportunities and barriers in these markets and explored the pathways to commercial acceptance. The study concludes that the residential, automotive, and other warmer temperature applications are not likely to become viable markets for the CCAR technology.

Table 6–9. Secondary Market Opportunities for CCAR Technology

Source: Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. 21.

Economic Analysis

The economic analysis portion of the study provided sufficient information about the model, assumptions, and data to make it easy to follow and replicate. Two scenarios were evaluated: a conservative base case and alternative “optimal” scenario. The optimal scenario was said to be consistent with the market studies and input from food processing and refrigeration industry experts, making it clear that the base case is conservative.

The study set a time period of 2002–2016 over which to forecast likely economic benefits. Like the RTI and Ehlen case studies, this case study separately identified benefits estimated to accrue directly to the joint venture partners and those estimated to accrue more broadly. Also like those cases, this case study applied a counterfactual analysis in deciding how to attribute estimated benefits from the CCAR technology to ATP.

Table 6–10 shows a summary of projected base case cash flows for application of the CCAR technology in the primary market, food processing. The contribution of each of the four different types of benefits within this market area can be seen.

Table 6–11 shows three estimated measures of public returns from ATP’s investment in CCAR development: net present value (NPV), internal rate of return (IRR), and benefit cost ratio. Discounted at a 7% rate, the NPV was estimated at $459 million. The social return on total investment was not estimated. Because the study concluded that the technology would not have been developed without ATP assistance, the estimated benefits used to calculate public returns are presumably the same as would be used in calculating social benefits, but the costs presumably would differ.

Table 6–10. Base Case Cash Flows from Improved Quality, Yield, and Production Rates and from Reduced Refrigeration Costs from Application of the CCAR Technology for Food Processing