Overview of the ATP

The ATP fosters technological advances through research and development that are expected eventually to increase the productivity and competitiveness of U.S. businesses, provide consumers with better and lower-cost products and services, and increase high-wage employment in the U.S. Businesses conceive, propose, and carry out the research projects that are cost-shared by the ATP. The multi-year ATP awards are made to individual companies and to joint research ventures -- comprised of two or more companies, often in combination with universities and research laboratories. A highly competitive peer review process is used to select award recipients. ⁽¹⁾

Unlike most other federal science and technology programs, the ATP is not a customer for the results of the technology development projects it funds. The ultimate outcomes are the results of marketplace interplay of demand and supply forces. But to improve the chances that the technologies developed will move out of the laboratory and into commercial use, the ATP's peer review process assesses not only the scientific and technical merit of proposals, but also the strength of proposed pathways to market and applicant commitment to follow through with commercialization plans, if the research is successful.

The ATP relies on the presence of expected private returns to induce companies to be willing to cost-share the research with the ATP and subsequently to carry out commercial development of the new technology with private capital. But it selects projects for awards for which it thinks the potential social rate of return (the return to the nation) far exceeds the private rate of return on investment (the return to the award-recipient innovators). Furthermore, the ATP seeks to select those projects for which the expected private return on investment is inadequate in the face of difficult technical challenges and appropriability problems to cause the private sector to be willing and able alone to do the project at all, or to do it within the critical time period, or to the scale/scope necessary to realize the full potential benefits.

The ATP does not fund streams of basic research in the traditional mode of the U.S. National Science Foundation. It is not a procurer of mission-related applied research in the mode of the U.S. Defense Department or the U.S. Department of Energy. It funds research; not product development. It operates in the middle ground between basic research and product development, with the goal of seeing commercial activity stem from the research.

The ATP held its first competition in 1990, and to date has funded a total of 352 projects, involving more than 800 participants (not including subcontractors and informal partners and collaborators). The projects entail approximately $2.3 billion of research, of which industry has committed to covering slightly more than half, and ATP the remainder. The earlier funded projects are now completing the research phase and an increasing number of them are moving into the commercialization phase. However, at this time, most of the funded projects are still in the research phase.

Why Evaluate?

The ATP initiated evaluation at the outset of the program; first, to develop a management tool to help the program meet its mission and operate more efficiently; and, second, to meet the many external requirements and requests for ATP program results. Demand for performance measures of the ATP is intense. Indeed, the ATP may be the most examined program relative to its budget size of any government program to date! Many requests are driven by policy issues. Specific requests for evaluation results come frequently from individual members of Congress and their staff, from Congressional subcommittees, the General Accounting Office, the Executive Office of the President, the Office of Management and Budget, the Office of Inspector General, the Press, think tanks, industry groups, and others. Title II of the American Technology Preeminence Act of 1991 (P.L. 102-245, enacted in 1992) directed that a comprehensive report on the results of the ATP be submitted to each House of the Congress and the President not later than 1996. (This report, The Advanced Technology Program: A Progress Report on the Impacts of an Industry-Government Technology Partnership, was delivered in April 1996.) In addition, the ATP, like other federal programs, is subject to the evaluation requirements of the 1993 Government Performance and Results Act (GPRA) . The GPRA resulted from a bipartisan effort to improve accountability, productivity, and effectiveness of Federal programs through strategic planning, goal setting, and performance assessment. The ATP/NIST, like other Federal government agencies, is developing assessment plans and techniques, and carrying out evaluation studies in compliance with the GPRA. The ATP receives many inquiries about its evaluation tools and methodologies from other agencies who are attempting to comply with the GPRA requirements. The ATP's data collection and methods of analysis were cited recently by the U.S. House Science Committee as a guide for programs authorized under the Small Business Technology Transfer Program Reauthorization Act of 1997 (HR 2429, 1997).

What and When to Measure

Because ATP's mission is economic in nature, its evaluation plan emphasizes the economic impacts of the program. However, there are a number of legislated requirements and constraints to the program that condition what is measured. For example, only projects that entail high-risk research that presses the state of the art and adds significantly to the scientific knowledge base are considered acceptable mechanisms for seeking the desired economic growth via the ATP. Hence, one aspect of the ATP's evaluation plan concerns the strength of purely scientific/technical contributions of funded research. Because the ATP is charged with promoting the formation of research joint ventures, this is similarly a topic of keen interest for ATP's evaluation plan. In addition, from the standpoint of designing and operating such a program, there is much to learn about underlying cause-and-effect relationships that collectively determine long-run outcomes. ATP's evaluation plan seeks to advance that understanding and to develop better models for capturing diverse and complex effects of the program.

With respect to measuring economic impacts, our approach distinguishes between initial impacts resulting from the commercial activities of the awardees and subsequent impacts resulting from spillover effects. Since it is legislated that U.S. businesses propose, lead, and cost-share the research, and that development and commercialization of technology be accelerated, one aspect of the ATP's evaluation plan concerns the commercial progress of award recipients in bringing goods and services based on the technologies to market. Moreover, since the program is focused on providing national benefit, the evaluation plan is concerned with spillover effects -- including market spillovers, knowledge spillovers, and network spillovers -- those effects that extend beyond the direct awardee innovators. Spillovers cause social returns and private returns to diverge. Figure 1 provides a conceptual illustration of private and social returns to R&D, taking into account pure market spillovers, pure knowledge spillovers, and an interaction of the two (Jaffe, 1996).

____________________
Jaffe, 1996

Main Components of ATP's Evaluation Program

To square the often urgent demands in the short run for evaluation results with the reality that patience is required to realize and validate empirically long-run program outcomes, the ATP has adopted a time-dimensioned, multi-faceted evaluation strategy. Its main components include (1) descriptive (statistical) profiling of applicants, projects, participants, technologies, and target applications; (2) progress measures derived principally from surveys and ATP's "Business Reporting System;" (3) real-time monitoring of project developments by ATP's staff; (4) case studies that focus in depth on project developments and results over defined periods; (5) projections of expected long-term impacts; (6) methodological research to improve the tools of long-term evaluation encompassing spillover effects; (7) special-issues studies to inform program structure, operations, and evaluation; and, ultimately, (8) empirical assessment of long-term project and program outcomes.

Overview of ATP's Evaluation Accomplishments through September 1997

Evaluation Planning

ATP's evaluation plan was developed by its economists, in consultation with leading economists in the field. We hold periodic workshops to obtain feedback on the approach and plans, to review work in progress, and to solicit advice on future directions. (Workshops on ATP's evaluation, co-chaired by Professor Zvi Griliches of Harvard University, were held at NIST in December 1994, September 1995, and January 1997; and at the National Bureau of Economic Research in August 1997.) Approximately at six-month intervals, the ATP conducts economic review boards to consider proposed evaluation studies against the backdrop of its plan. Guidelines are provided as a resource to evaluation researchers interested in proposing studies (Ruegg, 1996).

Systematic Data Collection and Analysis

ATP's "Applicant Database" and "Awards Database" provide answers to numerous questions about what, where, and whom ATP is funding. These databases provide descriptive information important to evaluation. Table 1 shows summary statistics from the databases.

____________________
ATP applicant and award databases.

An integrated set of databases, called the "Business Reporting System" is a key component of ATP's data compilation in support of project evaluation. These data, compiled by electronic survey of individual project participants, are used to track the evolution of projects towards achieving their business and economic goals. The Business Reporting System consists of five parts: An initial report on planned application areas for the technology and planned strategies for eventual commercialization; quarterly short reports on major interim business developments; an annual report on progress towards implementing the commercialization strategies and on short-term economic impacts of the projects, including, but not limited to, early sales revenues, shortening of R&D cycles, collaboration effects, intellectual property creation, and early job creation; a close-out report which completes the reporting of developments during the course of the research project and updates future plans for commercialization; and a post-project report which updates progress and is to be administered three times (every other year) over the period following project completion. A recent ATP report draws from this extensive database to summarize project-, company-, and application-specific progress of over 200 projects added to the reporting system between 1993 and 1995 (Powell, October 1997, draft). Those project participants in the reporting system had identified more than 1,000 applications of the technologies under development and provided commercialization plans for almost 800 of the applications by the end of 1996. Figure 2 shows for a subset of projects in the system the reported progress towards commercialization of their technologies developed with ATP support. The Business Reporting System database has considerable potential for serving the needs of evaluation research, and we expect to make extensive use of it.

____________________
Business Progress Reports for 778 applications being pursued by 375 companies in 207 ATP projects funded 1993-1995. (Powell, December 1997)

Special Surveys of ATP Participants

The ATP has sponsored two broad surveys of funded companies by third-party contractors. These contractors used structured telephone interviews to assess the progress of early projects not included in the business reporting system, and the satisfaction of participants with the program (Solomon, 1993, and Silber, 1996). An example of survey results is shown in Figure 3. Seventy percent of ATP project respondents reported that they likely would not have developed the technology at all without the ATP award. Of those who said that they likely would have proceeded without the ATP, over 90% reported that their level of effort, goals, and rate of progress would have been substantially lower without the ATP award (Silber, 1996). Similar results have been reported by the U.S. General Accounting Office (GAO, 1996), and are substantiated by the recent analysis of Business Reporting System data (Powell, October 1997, draft). These findings suggest that ATP funding is leveraging private capital rather than displacing it, and that the program is accelerating the development and commercialization of technology as it is charged to do -- both issues of critical importance to the ATP. Further research on these issues is underway.

____________________
Silber, 1996

A recently completed survey of 28 projects funded in 1991 focused on the topic of research cycle-time reduction and how it translates into acceleration of commercialization, increased project benefits, and other effects. Ninety-six percent of the companies reported that it was very important for their companies to reduce cycle time and that their participation in the ATP reduced their research cycle time by a median value of 50% or three years. Eighty-six percent of the companies expected the reductions in research time to flow through to later stages in the technology development and commercialization cycle. Many of the companies gave ballpark estimates of the value of reducing research time by just one year which ranged from one million dollars to "billions." Interestingly, the companies reported that time savings in the project were carried over to other technology development projects. These findings suggest that the ATP may be fostering institutionalized practices that extend acceleration of technology development and commercialization beyond the direct projects funded (Laidlaw, 1997).

Microeconomic Case Studies

The earliest case studies of ATP projects focused on research efficiency in on-going joint ventures; several of these have now been updated. One of these, " Early Stage Impacts of the Printed Wiring Board Joint Venture," (Link, October 1997, draft) reported research savings from collaboration on the project on the order of $35.5 million for the part of the research that the member companies thought they would have pursued, though at a slower pace, without the ATP. (The research cost savings and accelerated progress are in addition to the newly gained technical capabilities from the research they would not have pursued at all without the project.) In addition, member companies reported the beginning of a translation of the new capabilities into valuable productivity improvements. With respect to generating knowledge spillover benefits to the rest of the industry, the joint venture members contributed over 200 research paper presentations, two of which received awards for Best Paper of Conference.

Another set of case studies underway compares ATP-funded tissue engineering technologies with current approaches ("defender technologies") in their first expected application areas: stem cell transplants, cancer diagnosis, heart transplants, orthopedics, and treatment of diabetes (RTI, October 1997, draft). Medical benefits took into account per-patient changes in quality-adjusted life-years. Based on conservative assumptions, and taking into account just one type of application per technology of the multi-application technologies, the preliminary estimates are that ATP's investment will yield more than $35 billion in expected net present value benefits to the nation. It is expected that the net benefits would be even higher if additional application areas were taken into account, since the background research would already be done.

Another set of case studies nearing completion assesses the economic significance of ATP projects aimed at improving the performance of material surfaces under harsh conditions (Faucett, October 1997, draft). Surface wear and tear is a costly problem in everything from machine tools to artificial hip sockets -- problems that ATP funded projects are attempting to address with innovative technological solutions. Results of the study are not yet available. Other case studies are also underway.

Macroeconomic Impact Projections

While microeconomic case studies provide detailed analyses of projects, macroeconomic models can provide projections of national economic impact of ATP-supported technologies. The ATP has experimented with a regional econometric model and is examining others. A recent study used a regional econometric model to examine the economic impact of an ATP-funded project to improve control of dimensional variation of stamped metal parts in the production of automobile bodies and other discrete-parts manufacturing (CONSAD, 1997). The study projected increases in Gross Domestic Product of at least $3 billion and employment gains of several hundred thousand jobs. These estimates took into account inter-industry effects of a predicted small increase in market share for U.S. automobile producers attributable to a quality improvement resulting from the ATP project.

The ATP, in collaboration with another NIST office, is using econometric modeling to analyze the aggregate impacts of several other ATP projects. We have concluded that the approach is suitable for evaluating the macroeconomic impact of certain kinds of ATP projects.

Statistical Analysis of Research Joint Ventures

An extensive empirical analysis and comparison of joint ventures funded by the ATP with non-ATP joint ventures is now in preparation (Vonortas, October 1997, draft). Preliminary findings show that the two sets of joint ventures appear complementary. Membership in the ATP-funded joint ventures appears to involve more small and medium-sized enterprises than membership in the non-ATP ventures. Moreover, ATP-funded joint ventures have much lower participation by foreign-owned companies and a higher participation of universities than the non-ATP ventures. The two groups have had about the same level of participation by nonprofits and government organizations. The preliminary study also concluded that research funded by the ATP enhances network spillovers and may add to the structural flexibility of the economy. Both joint venture groups experienced a positive effect on firm incentives to collaborate again. Interestingly, the pairs of industries brought together in ATP joint ventures demonstrated the least "relatedness," suggesting more research experimentation. Also, the study concluded that firms participate in research joint ventures to mitigate early-stage technological and market uncertainty in areas with strong growth potential and to create "technology options" for future exploitation.

Learning from Other Programs

The ATP is interested in taking advantage of the insights gained by researchers evaluating counterpart programs to the ATP. As background to this effort, the ATP will release soon a report comparing principal features of the ATP and similar programs of other nations (Chang, October 1997, draft). An ATP research project currently in the early stages is testing an evaluation model with data drawn from technology development programs of several countries.

International Conference on Evaluation

The ATP, in conjunction with the NBER, recently announced an "International Conference on the Economic Evaluation of Technological Change," to be held at the Georgetown University Conference Center, Washington, D.C., June 15-16, 1998. The conference will combine small group workshops with larger plenary sessions. The common element of all papers and presentations will be an emphasis on the empirical assessment of the effects of government investment in science and technology in advanced industrial economies. The goal of the conference is to advance the state-of-the-art of evaluating the impacts of government investments in high-risk research to develop enabling technologies of the type funded by the ATP. The meeting will bring together government R&D policy makers, analysts, and R&D managers, with academic and other researchers studying technology evaluation issues. It is expected that the presentations will facilitate comparisons of evaluation methods, and practices in different countries. Registration information can be obtained on-line from the ATP website at http:\www.atp.nist.gov, or by writing the conference organizers at the addresses given below.

or

Evaluation: An On-Going Activity

In addition to the studies mentioned above, the ATP has a number of other evaluation studies underway. Research topics include spillover pathways and models; research collaborations; inter- and intra-industry diffusion mechanisms, patterns, and rates; development of new and improved qualitative and quantitative models for measuring economic impacts of publicly funded, privately executed technological advances; technology financing issues; and organizational issues affecting project structure and participants.

Consistent with the advice we have received from experts in the field (e.g., Mansfield, 1996), we view the estimation of social and private returns as a dynamic and on-going process. We understand and acknowledge that the rough quantitative projections that are possible now based on limited information will become better informed as commercialization and diffusion activities progress. By tracking developments as they unfold, we expect over time to be able to reduce the estimating errors and extend the scope of analysis. Over the next five years, the ATP expects to contribute significantly to the body of work on technology impact assessment.

Acknowledgements

Appreciation is extended to Dr. Harold Marshall, Ms. Connie Chang, and Ms. Jeanne Powell, all economists at NIST, for their helpful reviews of the draft manuscript.

References

Chang, Connie. A Multi-Country, Binational Comparison of Government R&D Programs Similar to the U.S. Advanced Technology Program. Gaithersburg, MD: NISTIR in preparation, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, October 1997.

CONSAD Research Corp. Advanced Technology Program A Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing. Gaithersburg, MD: NIST Grant/Contract Report 97-709, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, March 1997.

Jack Faucett Associates. Returns from ATP-Funded Innovation: Engineered Surfaces , Gaithersburg, MD: NIST Grant/Contract Report in preparation, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, October 1997.

Jaffe, Adam. .htmEconomic Analysis of Research Spillovers: Implications for the Advanced Technology Program. Gaithersburg, MD: NIST Grant/Contract Report 97-708, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, December 1996.

Laidlaw, Frances Jean. Acceleration of Technology Development By the Advanced Technology Program: The Experience of 28 Projects Funded in 1991. Gaithersburg, MD: NISTIR-6047, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, September 1997.

Link, Albert N. Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End. Gaithersburg, MD: NIST Grant/ Contract Report in preparation, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, October 1997.

Mansfield, Edwin. Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities. Gaithersburg, MD: NIST Grant/ Contract Report, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, January 1996.

National Institute of Standards and Technology. Advanced Technology Program Proposal Preparation Kit. Gaithersburg, MD, U.S. Department of Commerce, Technology Administration, December 1997.

National Institute of Standards and Technology. The Advanced Technology Program: A Progress Report on the Impacts of an Industry-Government Technology Partnership. Gaithersburg, MD: NIST-ATP-96-2, U.S. Department of Commerce, Technology Administration, April 1996.

Powell, Jeanne. Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report for Projects Funded 1993-1995. Gaithersburg, MD: NISTIR 6098, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, December 1997.

Research Triangle Institute (RTI), Center for Economic Research. A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies; Preliminary Applications to Tissue Engineering Projects Funded from 1990 to 1996. Gaithersburg, MD: NIST Grant/Contract Report in preparation, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, October 1997.

Ruegg, Rosalie. Guidelines for Economic Evaluation of the Advanced Technology Program. Gaithersburg, MD: NISTIR-5896, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, November 1996.

Silber & Associates. Survey of Advanced Technology Program 1990-1992 Awardees: Company Opinion about the ATP and Its Early Effects. Gaithersburg, MD: NIST Grant/Contract Report 97-707, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology,January 1996.

Solomon Associates. The Advanced Technology Program; An Assessment of Short-Term Impacts: First Competition Participants. Gaithersburg, MD: NIST Grant/Contract Report, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, February 1993.

U.S. Congress, House Science Committee, HR 2429, Small Business Technology Transfer Program Reauthorization Act of 1997.

U.S. General Accounting Office. Performance Measurement: The Advanced Technology Program and Private Sector Funding. Washington, DC: GAO/RCED-96-47, Code 307723, 1996.

Vonortas, Nicholas S. The Effectiveness of Research Joint Ventures Sponsored by ATP. Gaithersburg, MD: NIST Grant/Contract Report in preparation, U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, October 1997.

____________________

1. Application guidelines and selection criteria are covered in "Advanced Technology Program Proposal Preparation Kit, November 1996," which is updated and reissued periodically.

Date created: June 1996
Last updated: April 12, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)