 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
|
A Toolkit
for Evaluating Public R&D Investment Executive SummaryEvaluation is a powerful tool for decision makers, but only if it is correctly structured, managed, and applied. Among federal and state agencies interested in science and technology, the Advanced Technology Program (ATP), located in the U.S. Department of Commerce’s National Institute of Standards and Technology (NIST), has emerged as a leader in the effective use and development of evaluative tools. Over its first decade, ATP followed a multi-faceted approach to evaluation, providing a mosaic of findings about how the program works and its impacts. This report assembles a large body of ATP’s evaluation studies into a coherent framework, making the studies more accessible and understandable to a diverse audience. An expected benefit is better utilization of past evaluation and increased efficiency and effectiveness in planning future evaluation. In effect, the report provides an evaluation “toolkit” for ATP that will also be useful to others who operate public technology programs. The toolkit provides an evaluation framework; a directory of evaluation methods, tools, techniques, principles, explanatory information, and best practices; an account of ATP’s use of evaluation models and methods over its first decade as revealed in a body of 45 selected studies; a cross-cutting compendium of findings; and recommendations for future work. The report addresses the science, craft, and art of evaluation in the context of ATP. It shows how a program established in a climate of political and conceptual debate can use evaluation techniques to answer questions about its fundamental rationale, design features, and economic impacts. Part I provides a general framework for evaluation, discussing evaluation fundamentals and methods, best practices, and an evaluation logic model to describe ATP’s evaluation program. Part II demonstrates the use of evaluation methods by drawing on ATP evaluation studies. Part III presents the emerging body of knowledge from studies of ATP—knowledge about firm behavior, collaboration, spillover effects, interfaces with state and international technology programs, ATP’s performance at large, and knowledge about evaluation itself. Part III also presents the authors’ conclusions and recommendations. Other features include a glossary of terms, methods bibliography, and a quick reference guide to evaluation models and methods, ATP studies cited, and study findings on program impacts. Evaluation Underpinnings For a public sector program like ATP, evaluation seeks to measure change and to determine if the change is attributable to program intervention. An effective evaluation program should investigate change in terms of a program’s mission-driven goals, and should compare its findings against intended results. As a point of departure, this report starts with a generic logic model of program evaluation, depicted below, and fleshes out the model using ATP as illustration. The report summarizes the major analytical themes economists and others use to explain the rationale for ATP: (1) Global economic competition is increasingly driven by technological advance; (2) enabling technologies tend to generate large spillovers; (3) high level of technical risks contribute to an R&D funding gap in the private sector; (4) many advanced technological development projects require multi-disciplinary and multi-organizational collaborative efforts; and (5) the nation’s capacity for economic competitiveness and prosperity depends in large part on its innovative capacity, which can be strengthened through publicprivate partnerships. ATP’s evaluation program has emphasized modeling its underlying program theory—exploring basic concepts, developing underlying causal maps, developing and refining analysis models, and investigating paths connecting program activities to intended impacts. Findings from ATP’s studies in turn have shaped its program and evaluation design in numerous ways. Multi-Faceted Methodological Approach Evaluators use a variety of evaluative methods, each with its advantages, disadvantages, and specialized purposes. In a multi-faceted approach, like that used by ATP, methods are chosen for their appropriateness to the question at hand, to cost and administrative feasibility, and to a purposeful mixture of methodological paradigms. Three dominant characteristics of ATP’s evaluation program have been the care with which methods and techniques have been matched to the questions being posed, the evolution toward more rigorous tests of causal relationships between ATP activities and observed outcomes, and the development of new tools when existing tools were not up to the task. The result is an extensive and increasingly sophisticated toolkit of methodologies available to evaluate ATP and other technology programs. The figure below depicts the major methods used by ATP over its first decade, and the changing intensity of their use over time. Forexample, the use of case study increased from 2 to 4 to 10 between the period 1991–1995, 1996–1999, and 2000, respectively. Generic Evaluation Logic Model
Note: The dynamics by which the transformation among these various stages occur are often complex and are themselves the subject of evaluation. An Emerging Body of Findings Throughout its history ATP has had to demonstrate that its operations added to, rather than displaced, the actions of the private sector in assembling the capital necessary to nurture high-risk, enabling technological innovations. It also has had to prove that ATP assistance produces economic benefits that extend beyond the direct recipients of ATP awards to generate broad benefits for the nation. Evaluation has provided descriptive and analytical information on program recipients and program outputs to ATP and NIST officials, to key executive and congressional decision makers, and to other stakeholders, including the general public. The body of evaluative work conducted over ATP’s first decade has answered central questions arising from ATP’s mission. ATP’s Evolving Use of Methods Over its First Decade *These 81 methods are employed in the 45 ATP studies commissioned between 1990 and 2000 that are examined in this report. A crosscutting analysis of the evaluation studies reviewed revealed much information that bears directly on ATP’s mission-driven goals. This analysis is organized around the following major themes: (1) firm/industry effects, (2) collaboration effects, (3) spillover effects, (4) interfaces and comparisons with other programs, and (5) measures of overall ATP performance, including portfolio analysis, social returns on investment, and impacts on competitiveness. Taken as a body of work, these studies have also contributed to enhanced understanding of the dynamics of the U.S. innovation system, particularly the characteristics of productive R&D relationships between the public and private sectors. Firm/Industry Effects Findings on private firm effects, drawn from 13 studies, indicated that ATP substantially expanded and enhanced the R&D activities of the companies examined. The studies provided a growing body of evidence that ATP funding is complementary to, not a substitute for, private sources of R&D funds. They indicated that ATP funding leverages and accelerates R&D, refocuses R&D on more technically challenging problems and enabling platforms of technologies, and fills a significant funding gap. One study concluded that the median timesavings per project was three years and the median economic value to the company per year saved was $5–$6 million. Two other studies estimated significant program-induced increases in patenting by ATP award recipients, indicating a positive impact of ATP on firm research productivity. With regard to the participation of small firms in ATP, the research showed robust participation rates and strong project performance relative to companies of larger size. Collaboration Effects The report drew findings on collaboration from 10 studies. One recurring conclusion was that there are high rates of collaboration in ATP projects, including formal joint venture members and extending strongly to single-applicant companies. For example, 84% of the first 50 completed projects entailed collaborative relationships, ranging from R&D partnerships with other firms, universities, and non-profit labs, to alliances with other firms to pursue commercialization. These studies found that ATP successfully encouraged applicants to propose projects entailing collaboration, frequently with entirely new partners. Collaborations of firms with universities was a topic of several of the studies on collaboration. Findings were that collaborations with universities were frequent and that they enhanced the research capabilities of the firms and provided an avenue of knowledge diffusion from and through the universities. Interestingly, studies also found that the collaborations were frequently fluid, with changes among collaborators occurring during a project’s life cycle. Some of these changes may be positive, keeping true to ATP criteria, while others may represent deviations, such as the loss of key participants or a retreat from the more challenging research goals, requiring ATP managerial intervention. The studies suggest that by monitoring projects throughout their lives, ATP is able to respond to and manage change. According to a study of joint venture participants, ATP contributes to joint venture success by: (1) accelerating the development of high risk technologies, (2) increasing project stability, (3) getting projects through particularly difficult periods in their life cycles, (4) overcoming barriers to collaboration, and (5) increasing up-front planning. Project participants identified specific benefits (particularly a positive effect on creativity), and costs (primarily increased administrative burden) associated with collaboration. Almost all project participants involved in collaborative arrangements indicated that their experience with ATP has stimulated them to plan additional collaborations. Among factors important to the success of collaborative relationships, the study corroborated other work that found establishing an environment of trust to be critical. Spillover Effects The concept of economic spillovers occupies a central place in the case for a public sector program like ATP and has helped shape many of ATP’s program design features. Findings from 10 of the studies increased understanding of ATP’s success in generating spillovers. The studies provided considerable evidence that ATP-funded projects generate outputs—publications, patents, patent citations, collaborative linkages, and products—that will potentially lead to knowledge and market spillovers. The potential of network spillovers was also identified, but not yet measured. One study concluded that ATP selects projects with attributes conducive to generating large knowledge spillover effects. Those attributes included linkages to other organizations, and a positive attitude of award winners toward information sharing. Several studies concluded that the degree to which a funded company is embedded in organizational networks is a major factor in knowledge spillover potential. Study results also indicated that ATP selects projects whose firms have more extensive ties to other businesses, and, hence, are better positioned to realize commercial success and related market spillovers. In the studies examined, quantitative estimation of the economic value of spillover benefits was limited to market spillovers. Where estimated, market spillover benefits appeared large, and far in excess of private benefits. Among the body of work examined, none of the studies estimated the economic value of both market spillovers and knowledge spillovers. Interfaces with State Programs and Comparison with Counterpart Programs Abroad This report draws on five studies for data on the interactions between ATP and state programs and on ATP-counterpart programs in other countries. One study’s major conclusion, based on analysis of existing state technology programs, was that state technology programs span the research and development continuum, but cluster around the downstream applied/commercialization segment rather than the upstream research segment of the continuum. The study found that ATP, in contrast, centers its activities on technical challenges, supporting work primarily in the concept and development phases. A collection of case studies highlighted the possibilities of firms combining support from both ATP and state government programs, and illustrated how ATP and the state programs can augment one another. With regard to counterpart programs in other countries, one study offers a framework for standardizing the comparison of ATP with foreign counterpart programs. This systematic approach has helped ATP meet its mandated requirement to test for eligibility of foreign-owned companies for awards and has allowed ATP to learn from the experience of other programs. Overall ATP Performance Thirteen studies provided findings on ATP’s impact on national industrial competitiveness and the national capacity to innovate, its ability to deal appropriately with failed projects, its contribution to social benefits, and its overall effectiveness. Prospective case studies provided evidence that the benefits of the program far exceed its costs. These studies collectively attributed to ATP more than $15 billion in expected present value social benefits from just a few projects, much greater than the total amount spent by the program. As expected, not all of the projects are strong performers, but several years after project end an estimated 16% of completed ATP-funded projects showed strong progress toward creating and disseminating knowledge and commercializing projects and processes, and another 26% also showed substantial progress. Five to 6% of all funded projects failed to start or were terminated prior to completion for a variety of reasons. In a major independent assessment, the National Research Council concluded that ATP is effectively meeting its legislative goals. Recommendations for Future Directions This report concludes by proposing future directions for ATP’s evaluation program, taking into account stakeholder questions, gaps in coverage, past accomplishments, and promising research opportunities. The authors provide 10 recommendations, in no particular rank order, as follows:
In sum, evaluation has provided an objective analytical and empirical basis for assessing ATP’s operations and impacts during its first decade of operations. Cumulatively, these evaluations highlight the value of applying multiple evaluation methods to complex problems, building a body of credible evidence over time that ATP is achieving its objectives. Main topics covered in the report are highlighted below.
*Some studies used multiple methods. Not all studies are referenced in each chapter. Return to Table of Contents or go to Acronyms and Abbreviations. Date created: July 13,
2004 |
|||||||||||||||||||||||||||||||||||||||||||||
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov. NIST is an agency of the U.S. Commerce Department |
 |
