A Toolkit for Evaluating Public R&D Investment Models, Methods, and Findings from ATP's First Decade

CHAPTER 3: ATP's Evaluation Program

Background: Evaluation Drivers

In large measure, the development of a strong evaluation program by ATP was internally driven. ATP’s standing as an experimental undertaking, established by the Omnibus Trade and Competiveness Act of 1988, together with the perspective of its first director, ⁵⁷ contributed to an environment of curiosity and learning. Evaluation was seen as a tool of learning. A small amount of ATP’s initial 1990 budget was set aside by program officials to fund rudimentary evaluation activities. This interest in evaluation continued to grow over time. A time line of events and developments critical to ATP and its evaluation program during its first decade is provided below:

• 1988 Omnibus Trade and Competitiveness Act (P.L. 100–418) authorizes ATP.
• 1990 ATP receives its first budget of $10 million.
• 1992 American Technology Preeminence Act (P.L. 102–245) amends P.L. 100–418.
• 1993 Government Performance and Results Act (GPRA) requires evaluation.
• 1993 Scale up of ATP under consideration.
• 1996 Progress report on ATP’s impacts due to Congress.
• 1997 Secretary of Commerce Daley orders a 60-day review of ATP.
• 1998 Senate Report 105–234 requests independent assessment of ATP.

The American Technology Preeminence Act of 1991, enacted in 1992, amended the legislation establishing ATP, and directed that:

The Secretary [of the Department of Commerce] shall, not later than 4 years after the date of enactment of this Act, submit to each House of the Congress and the President a comprehensive report on the results of the Advanced Technology Program ... including any activities in the areas of high-resolution information systems, advanced manufacturing technology, and advanced materials.

Knowing it would have to report on results no later than 1996 provided another reason for ATP to strive to build its evaluation capabilities.

Because of these and other forces at work, ATP was able to take passage of the GPRA in stride. The GPRA’s requirement for the reporting of particular types of performance metrics influenced ATP to produce those measures. Measures that ATP considered important, such as spillover measures and collaboration results, seemed to fit GPRA requirements less well, while numbers of projects funded and completed, and other accomplishments that lent themselves to counting and trend lines, seemed to fit better. Therefore, ATP established the tracking of a set of performance metrics acceptable for GPRA reporting, while continuing its indepth, more complex economic and sociological evaluation studies that were highly informative of the workings and long-term performance of the program, though impossible to grasp in a single number or trend line.

With a scale up of ATP under consideration in 1993 and a change in the makeup of Congress in 1994, the program became the object of considerable debate and intense scrutiny. The General Accounting Office and the Office of the Inspector General carried out a number of studies of the program, and members of Congress asked the program to address many questions. In this environment, the importance of evaluation increased for ATP.

Preparation of the mandated report to Congress in 1996 also had a stimulating effect on further evaluation, providing a good opportunity to assess progress and identify shortcomings. Former Secretary of Commerce Daley’s direction to the program the next year to review certain aspects of its operation gave further impetus to evaluation.

More recently, in 1998, the U.S. Senate directed ATP to arrange for a well regarded organization with significant business and economic experience to conduct a comprehensive assessment of the program, analyzing how well it has performed against the goals established in the authorizing statute. This directive led to an extensive review of ATP from 1999 through 2001, by the National Academy of Sciences/National Research Council’s Board on Science, Technology, and Economic Policy, drawing on papers presented at academy-organized workshops and roundtables and the body of work on ATP.

ATP’s Evaluation Logic Model

In-depth knowledge of ATP’s structure, mission, operational mechanisms, program features, and intended impacts was essential to developing its evaluation program. Fleshing out the skeletal, generic logic model of Figure 2–1 to provide a specific logic model for ATP provides a useful framework for understanding how its evaluation program was formed. Figure 3–1, supplemented by Table 3–1, depicts an evaluation logic model for ATP.

Reading Figure 3–1 from top down, ATP began with a congressional goal and approach, which, stated in broadest terms, was to increase national prosperity and quality of life by providing funding for the development of new technologies. Faced with a suite of alternative public policy strategies for supporting science and technology, Congress adopted a public-private partnership program as one element of an overall strategy to meet its goal. Congress authorized establishment of ATP, defined its mission, and provided direction to the formulation of the new program’s operational mechanisms, features, and intended impacts, which were further elaborated by the U.S. Department of Commerce through the federal rulemaking process. Table 3–1 lists major goals specified in ATP’s mission and highlights some of the program’s more important operational mechanisms and features.

ATP was directed to increase the nation’s scientific and technical knowledge base, expand and accelerate development and commercialization of generic (referred to synonymously as “enabling”) technologies, promote collaborative R&D, refine manufacturing processes, and increase the competitiveness of U.S. firms. It was directed to generate broad-based benefits for the nation; that is, benefits extending beyond the relatively narrow population of award-recipient organizations. A constraint was included that the program should ensure the “appropriate” participation of small businesses.

Rather than using an outright, no-strings-attached grant as the award mechanism, ATP uses cooperative agreements to enter into cost-sharing arrangements with award recipients. Awarded funds can be applied only to approved costs of research. Projects are selected from proposals submitted to ATP that are peer reviewed against published selection criteria. Each award is for a specific project with both an R&D and a business/economics plan, and with well-defined goals and a limited duration.

There are two routes by which the program is designed to deliver impacts and achieve broad-based benefits: a direct route by which ATP award recipients and their collaborators accelerate development and commercialization of technologies that lead directly to private returns and market spillovers, and an indirect route by which publications, presentations, patents and other means of knowledge generation and dissemination lead to knowledge spillovers. Market and knowledge spillovers from the program are looked to as a primary means for broadening the impact of funded projects substantially beyond the direct award recipients.

Table 3–1. ATP’s Mission, Mechanisms, and Features

Figure 3–1. ATP’s Evaluation Logic Model

The middle tier of Figure 3–1 shows the program’s inputs, outputs, outcomes, and impacts. Program inputs derive from congressional appropriations that provide budgets for making awards, convening staff to administer the process, and providing for equipment, facilities, and other administrative costs. Principal outputs include the funded projects, collaborative relationships formed as a result of the program, publications, patents, models and algorithms, and prototype products and processes. Principal outcomes include sales of new and improved products, processes, and related services; productivity effects on firms; changes in firm size and industry size; a change in the propensity of firms and other organizations to collaborate; the spread of resulting knowledge through citations of publications and patents and by other means; and knowledge and market spillovers as others adopt the funded innovations. Longer-term impacts relate back to the broad societal goal that drove the program’s creation, including increased GDP, employment gains, improvements in the quality of life through improvements in the nation’s health, safety, and environment, and improved international competitiveness of U.S. industry. Impacts may also include an effect on the nation’s capacity to innovate. Figure 3–1 also indicates “process dynamics,” which refers to the transformations through which program inputs, outputs, outcomes, and impacts are linked. These transformations are complex, and there is much to learn about them.

The lower tier of Figure 3–1 ties the evaluation strategies and objectives to the program. Evaluation focuses on the inputs, outputs, outcomes, impacts, and process dynamics of a program. Evaluation objectives include tracking progress of funded projects, using, for example, indicator metrics; understanding process dynamics; estimating benefits and costs of projects and of the program overall; identifying the more difficult to measure effects; relating findings back to the program’s mission; and applying tests of success (discussed in the following section). Additional objectives include disseminating evaluation results and feeding results back to program administrators to improve the program and to policy makers to inform them and to meet reporting requirements. Evaluation methods and tools used to achieve these objectives include those presented in Chapter 2.

Conceptual Tests of ATP’s Success

One of ATP’s central missions is to produce broad-based economic benefits. This suggests a test of success stated primarily in economic terms, and is a major factor in ATP’s decision to press the use of economic methods of evaluation. However, ATP’s mission is complex and multidimensional, and a single test of success is inadequate.

The following four tests can help define ATP’s accomplishment of its central mission, provided they are applied after sufficient time has passed to offer a fair test: ⁵⁸

• Test 1: Has the portfolio of ATP-funded projects produced large net social benefits for the nation?
• Test 2: Has the portfolio of ATP- funded projects contributed to enhanced United States economic and technological competitiveness?
• Test 3: If test 1 is met, is a large share of the benefits attributable to ATP?
• Test 4: Regarding the distribution of net benefits, do they extend well beyond the direct ATP award recipients?

Additional criteria are needed to test for achievement of other supporting objectives while holding to program constraints. These other supporting objectives and constraints include building the scientific and technical knowledge base, fostering collaborative research, refining manufacturing, and ensuring appropriate participation of small businesses.

ATP’s Approach to Evaluation

Who Evaluates?

ATP’s approach has been to try to capture the advantages and avoid the disadvantages of relying solely on either in-house evaluations or outside contractors. The ATP formed a core in-house group responsible for planning, guiding, and monitoring its evaluation efforts, for establishing and maintaining certain databases needed both for evaluation and program management, and for carrying out studies that could best be performed in-house. ⁵⁹

ATP has also used a variety of outside contractors to carry out evaluation studies. It formed an early association with the National Bureau of Economic Research (NBER) to ensure that it obtained the services of leading evaluators, as well as to provide another level of independent review of the studies. In addition, it formed a panel to review proposed evaluation studies. ATP periodically held a roundtable of notable evaluators who were invited in to hear and comment on presentations of studies planned and completed. ATP obtained additional reviews of evaluation studies from outside reviewers. ATP cooperated with other program assessors, including assessors from the General Accounting Office (GAO), the Office of Inspector General (OIG), and the National Academy of Sciences (NAS), to provide evaluation materials and results from surveys.

This strategy of combining in-house capability with substantial support from outside evaluators was designed to keep ATP’s evaluations focused, relevant, and efficient while ensuring credibility and gaining multiple perspectives, talents, and experience.

A Variety of Methods Used in a Portfolio Approach

One strength of ATP’s evaluation program has been its strategy to use a variety of methods to evaluate program effects, choosing the best method for the task rather than focusing on a single method. Taking this multi-faceted approach, ATP has given more attention to some evaluation methods than to others during its first decade. Differential emphasis on the methods resulted from programmatic considerations including mission relevancy, the time delay before certain outputs and outcomes could result, and the need to respond to specific executive or legislative requests for information. Figure 3–2 suggests the timing and relative intensity with which the various evaluation methods have been applied to ATP.

Figure 3–2. Intensity of ATP’s Use of Evaluation Methods

*These 81 methods are employed in the 45 ATP studies commissioned between 1990 and 2000 that are examined in this report.

Building a Portfolio of Evaluation Studies

The sequence of evaluation studies commissioned by ATP over the past 10 years reveals how evaluation of the program evolved. Tables 3–2, 3–3, and 3–4 show the sequence of studies and indicate the principal method(s) used by each study. Two features are particularly notable: the increasing volume of studies as ATP ended its first decade and the increasing sophistication and greater focus of the studies funded. Starting with conceptual studies and surveys, ATP added case studies and econometric/statistical studies, and more recently undertook patent citation-tracing studies and network analysis.

Table 3–2. Select ATP Studies Commissioned and Completed, 1991–1995

*The paper was published independently of the ATP.

____________________
57 ATP’s first director, George Uriano, with his background combining scientific, administrative, and business expertise, played a key role in shaping ATP and its early evaluation effort. (For more information on Mr. Uriano’s background, see NIST Press Release 94–36, dated September 13, 1994.)

58 These are expanded from suggested tests put forth by R. Ruegg, “Assessment of the ATP,” in Charles W. Wessner, ed., The Advanced Technology Program, Challenges and Opportunity (Washington, DC: National Academy Press, 1999), p. 80.

59 ATP’s in-house evaluation staff is the Economic Assessment Office (EAO), whose primary focus is evaluation, but has other program responsibilities, such as serving on technology proposal selection boards and helping to oversee funded technology projects from an economics and business perspective.

Return to Table of Contents or go to Part II.

Date created: July 13, 2004 Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)