INTRODUCTION

ATP: A Partnership with Industry

The ATP attracts challenging, visionary projects with the potential to develop the technological foundations of new and improved products, processes, and even industries. The ATP partners with industry on this research, fostering collaborative efforts and sharing costs to bring down high technical risks and accelerate technology development and application. These are projects that industry in many cases will not undertake without ATP support, or will not develop in a timely manner when timing is critical in the highly competitive global market. The program funds only research, not product development. The ATP is managed by the National Institute of Standards and Technology, an agency of the Commerce Department's Technology Administration.

ATP awards are made on the basis of a rigorous competitive review, which considers the scientific and technical merit of each proposal and its potential benefits to the U.S. economy. The ATP issues a proposal preparation kit that presents and explains the selection criteria to prospective applicants and provides guidance on preparing proposals.² U.S. businesses conceive, plan, propose, and lead the projects. Government scientists and engineers who are expert in the relevant technology fields review all proposals for their technical merit. Business, industry, and economic experts review the proposals to judge their potential to deliver broadly based economic benefits to the nation—including large benefits extending beyond the innovator (the award recipient).

The ATP delivers benefits to the nation along two pathways: 1) a direct path by which the U.S. award recipient or innovator directly pursues commercialization of the newly developed technologies; and 2) an indirect path which relies on knowledge transfer from the innovator to others who in turn may use the knowledge for economic benefit. Either path may yield spillover benefits. The ATP looks to the direct path as a way to accelerate application of the technology by U.S. businesses. It looks to the indirect path as a means of achieving additional benefits, or benefits even if the award recipient fails to continue. The ATP's two-path approach to realizing national benefits offers advantages: one path may provide an avenue for benefits when the other does not, and both paths together may yield larger, accelerated benefits as compared to having a single route to impact.

Project Evaluation

The ATP, like other federal programs, is required by law to report on its performance.³ The ATP established its evaluation program soon after it began, even before evaluation was widely required by Congress. The Economic Assessment Office (EAO) of ATP plans and coordinates the evaluation of funded projects. It is assisted in this effort by leading university and consulting economists and others experienced in evaluation.

Performance is measured against the program's legislated mission. Emphasis is placed on attempting to measure benefits that accrue not only to the direct award recipients, but also to a wider population, i.e., spillover benefits. This emphasis reflects the fact the public funding covers part of the costs of these projects, and, therefore, a relevant question is how the broader public benefits from the expenditure.

This report constitutes one element of the EAO's multi-faceted evaluation plan: status reports. The purpose of status reports is to provide an interim assessment of the status of ATP-funded projects several years after they are completed. Although the ultimate success of the ATP depends on the long-run impacts of the entire portfolio of ATP projects, the performance-to-date of this partial portfolio provides some initial answers. This study contains an evaluation of 100 completed projects: the results of the 50 projects from the Status Report - Number 2, and the results and status reports of a second batch of 50 projects. These reports address the questions of: what has the public investment of $206 million in the 100 projects produced several years after completion of the research, and what is the outlook for continued progress?

Study Approach

From the moment that ATP funded its first group of 11 projects in the 1990 competition, program administrators, the administration, Congress, technology policymakers, industry, and others in this country and abroad were keenly interested in the outcome. But technology development and commercialization are lengthy processes, and it takes time to produce results.

As more ATP-funded projects are completed and move into the post-project period, sufficient time has elapsed for knowledge to be disseminated and progress to be made towards commercial goals. Thus, it is now possible to compile more complete aggregate portfolio statistics and analyze these statistics with regard to implications for overall program success.

At the core of this study are 50 mini-case studies covering each of the completed projects. Each of these briefly tells the project story, recounting its goals and challenges, describing the innovators and their respective roles, and assessing progress to date and the future outlook. Photographs illustrate many of the projects.
Although the particulars vary for each project, certain types of data are systematically collected for all of them. Consistent with ATP's mission, the evaluation focuses on collecting data related to the following dimensions of performance:

• Knowledge creation and dissemination, which is assessed using the following criteria: recognition by other organizations of a project's technical accomplishments; numbers of patents filed and granted; citations of patents by others; publications and presentations; collaborative relationships; and knowledge embodied in and disseminated through new products and processes.
• Commercialization progress, which is gauged in terms of the attraction of additional capital for continued pursuit of project goals, including resources provided by collaborative partners; entry into the market with products and services; employment changes at the small companies leading projects and other indicators of their growth; awards bestowed by other organizations for business accomplishments of project leaders; and the analyst's assessment of future outlook for the technology based on all the other information.

The approach is to provide, in an overview chapter, the aggregate statistics of interest across a set of 100 projects, such as the total number of patents and the percentage of projects whose technologies have been commercialized. In addition, the aggregate statistics are combined to produce composite project metrics for overall performance. The composite performance scores allow one to see at a glance the robustness of a project's progress towards its goals. Underlying the simple scores is a wealth of data.

Sources of Information

Data for the projects were collected from many sources: ATP project records; telephone interviews with company representatives; interviews with ATP project managers; company websites; the U.S. Patent and Trademark Office; in-depth project studies conducted by other analysts; academic, trade and business literature; news reports; filings at the Securities and Exchange Commission; and business research services, such as Dun and Bradstreet, Hoover's Online, Industry Network, and CorpTech. Each one of the individual project write-ups was reviewed for accuracy by the project's lead company and ATP staff.

Study Limitations and Future Directions

Since developments continue to unfold for most of these projects, the output measures for the cases may have changed significantly since the data were collected. The cases provide a snapshot of progress several years after the completion of the ATP-funded projects.

Although undertaken at different calendar dates, the reports are written within about the same interval of time after ATP funding ended. Yet, different points in each technology's life cycle may be captured, depending on the technology area. Information technology projects, for example, may be expected to be further along than advanced materials and chemical projects. Examined at a later time, there may be less (or more) difference in the accomplishments among projects in different technology areas.

This study tracks outputs leading to knowledge dissemination but it does not assess the actual commercialization efforts by others who acquire the knowledge. The tracking of commercialization efforts is limited to the direct path of impact (i.e., commercialization by the award recipients or innovators).

"Completed" and "Terminated" Projects Defined

Projects do not necessarily finish in the order funded. For one thing, they have different lengths, ranging from approximately two years to no more than five years. For another, they are required to file a final report with the ATP and have financial and other paperwork completed before project closeout. The financial closeout is done through the National Institute for Standards and Technology (NIST) Grants Office, which notifies the ATP that it considers the project completed. This study assesses the first 100 projects the Grants Office declared "completed."

Although most ATP projects reach completion, some are stopped short and are classified as "terminated." Some of these were announced as award winners but never officially started. Other projects got off the ground but were closed for various reasons with a substantial amount of the technical work still unfinished. These terminated projects are assessed according to the principal reasons they stopped before completion. They are treated in Appendix B. While the terminated projects are generally regarded as unsuccessful, some produced potentially useful outputs.

Report Organization

Chapter 1 provides a summary overview of the performance of the 100 completed projects as a group. It identifies some major outputs that appear useful as indicators of the degree of project success, and it uses these outputs in a prototype project performance rating system. A preview also notes some of the broad-based benefits that this portfolio of projects is producing and likely to produce. For additional background, the make-up of the portfolio of projects in terms of technologies, organizational structure, company size, and other features is provided.

The individual project reports of a second group of 50 are presented in Chapters 2 through 6 , and highlight major accomplishments and the outlook for continued progress. A detailed account of the project under review is given, with attention to technical and commercial goals and achievements, information about technology diffusion, and views about the role played by ATP funding. A performance rating is assigned to each project based on a four-star scoring system. The rating depends on the accomplishments of the project in creating and disseminating new scientific and technical knowledge and in making progress toward generating commercial benefits, as well as the outlook for continued progress.

Three appendices provide supporting information. Appendix A provides a listing of technical and commercial achievements of each completed project. Appendix B provides a discussion of the terminated projects throughout ATP's existence. Appendix C provides a list of the first 100 completed projects and the respective composite performance ratings. The list is sorted in descending order of performance rating, then by company name.

____________________
1. For an overview of the first 50 completed projects, see Performance of 50 Completed Projects, Status Report—Number 2, 2001. This study can be found on ATP's website (www.atp.nist.gov).

2. The current edition of the kit and other program materials may be obtained on ATP's website (www.atp.nist.gov), by e-mail (atp@nist.gov), by phone (1-800-ATP-Fund or 1-800-287-3863), or by mail (ATP, NIST, 100 Bureau Drive, Stop 4701, Gaithersburg, MD 20899-4701).

3. The Government Performance and Results Act (GPRA) is a legislative framework for requiring federal agencies to set strategic goals, measure performance, and report on the degree to which goals are met. An overview of the GPRA is provided in Appendix 1 of the General Accounting Office Executive Guide, Effectively Implementing the Government Performance and Results Act, GAO, Washington, D.C., GGD-96-118, 1996

Return to Table of Contents or go to next section of Status Report No. 3.

Date created: April 4, 2006
Last updated: July 5, 2006

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)