The ATP funds projects that are expected to make significant contributions to the scientific and technical knowledge base; lead to new technologies that the awardees will take into the marketplace using non-ATP sources of funding; and, in the long-run, yield substantial national economic benefits beyond those going directly to ATP-award recipients. The research projects that the ATP funds are typically 2 to 5 years in duration. The commercialization phase generally takes at least several years more. The widespread diffusion of the technology across and within industry sectors takes yet more time. The bottom line is that ATP's evaluation program must track the short-term results, as well as the medium-to long-term economic impacts for the nation.

Evaluation begins in the ATP at the very outset of program and project selection. When the ATP receives project proposals from U.S. businesses, it uses peer evaluation to score the proposals in terms of their scientific and technical merit and their business and economic merit. Once the funded projects are up and running, the ATP conducts real-time evaluations of project progress by monitoring the accomplishments of the participating organizations in carrying out their research and in planning for future commercialization. ATP's project management teams perform site visits and annual reviews. Their project oversight is facilitated by technical and business quarterly reports direct from the companies to the ATP on their research progress, as well as on their business-related progress. The business reporting system ⁽¹⁾, which tracks progress towards future applications of the technologies, is an essential component of ATP's present and future project and program evaluation, and enables empirical analysis of the portfolio of projects to be performed by researchers. The reporting system is standardized to facilitate statistical analysis, and it is electronically administered. To assess the opinions of participating organizations who are at the very heart of ATP's success or failure, the ATP from time-to-time commissions third-party surveys of the participating companies. Third-party surveys are one way for the ATP to learn what is working well and what needs attention. They also help to identify impacts of the ATP at the firm level. The "Silber Study⁽²⁾," for example, reports the results of such a survey. To gain a detailed understanding of the effects of the ATP at the firm level, the ATP also carries out project case studies. Case studies conducted at an early stage of a project may focus narrowly on changes experienced by the participating company or companies as a result of the ATP project; e.g., changes such as research efficiencies⁽³⁾ experienced by research joint ventures, or improvements in research cycle time⁽⁴⁾, or new business opportunities and growth in the company as a result of the ATP award. Other case studies explore the rate of adoption of the technology, and attempt to measure spillover benefits and costs⁽⁵⁾ that accrue to users of the technology beyond the ATP participating companies. Studies to increase the understanding of spillover mechanisms⁽⁶⁾ are critical to the evaluation of the ATP, and are also being carried out. A potential approach to projecting impacts from the firm level across the entire economy (e.g., national employment and industrial output measures) is to apply large-scale macroeconomic models paired with microeconomic project analysis. The ATP is currently experimenting with this approach, has released one study⁽⁷⁾ and expects soon to release another. The ATP is exploring additional approaches to measuring the impact of the ATP, and periodically convenes working sessions with the nation's leading economists to discuss evaluation models, results, and opportunities. A number of additional evaluation studies are underway and will be added to this site as they are completed.

1. For further information on ATP's Business Reporting System, please see Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report for Projects Funded 1993-1995, Jeanne Powell, ATP/NIST, November 1997.

2. See Survey of Advanced Technology Program 1990-1992 Awardees: Company Opion About the ATP and its Early Effects, Silber & Associates, January 1996.

3. For an example of a study on research efficiencies experienced by ATP-funded research joint ventures, please see ATP Early Stage Impacts of the Printed Wiring Board, Albert N. Link, Consulting Economist, University of North Carolina at Greensboro, November 1997.

4. For an example of a study on improvements in research cycle time in ATP-funded projects, please see Acceleration of Technology Development by the Advanced Technology Program: The Experience of 28 Projects Funded in 1991, Frances Jean Laidlaw, Industry Consultant, October 1997.

5. For an example of a study on the measuring spillover benefits and costs of ATP-funded projects, please see A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, Research Triangle Institute, Center for Economic Research, Research Triangle Park, NC, April 1998.

6. For a study on understanding spillover mechanisms, please see Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, Adam B. Jaffe, Brandeis University and National Review of Economic Research, December 1996.

7. For an example of a study that pairs large-scale macroeconomic models with microeconomic project analysis in evaluating impacts of ATP-funded projects, please see Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing, CONSAD Research Corporation, March 1997.

Return to Overview of the EAO

Date created: June 1996
Last updated: April 12, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)