Performance of 50 Completed ATP Projects Status Report - Number 2 NIST SP 950-2 Introduction

ATP: A Partnership with Industry
The ATP attracts challenging, visionary projects with the potential to develop the technological foundations of new and improved products, processes, and even industries. The ATP partners with industry on this research, fostering collaborative efforts and sharing costs to bring down high technical risks and accelerate technology development and application. These are projects that industry in many cases will not undertake without ATP support, or will not develop in a timely manner when timing is critical in the highly competitive global market. The program funds only research, not product development. The ATP is managed by the National Institute of Standards and Technology, an agency of the Commerce Department’s Technology Administration.

ATP awards are made on the basis of a rigorous competitive review which considers the scientific and technical merit of each proposal and its potential benefits to the U.S. economy. The ATP issues a proposal preparation kit that presents and explains the selection criteria to prospective applicants and provides guidance on preparing proposals.⁽¹⁾ U.S. businesses conceive, plan, propose, and lead the projects. Government scientists and engineers who are expert in the relevant technology fields review all proposals for their technical merit. Business, industry, and economic experts review the proposals to judge their potential to deliver broadly based economic benefits to the nation —including large benefits extending beyond the innovator (the award recipient).

The ATP delivers benefits to the nation along two pathways: 1) a direct path by which the U.S. award recipient or innovator directly pursues commercialization of the newly developed technologies; and 2) an indirect path which relies on knowledge transfer from the innovator to others who in turn may use the knowledge for economic benefit. Either path may yield spillover benefits. The ATP looks to the direct path as a way to accelerate application of the technology by U.S. businesses. It looks to the indirect path as a means of achieving additional benefits, or benefits even if the award recipient fails to continue. The ATP’s two-path approach to realizing national benefits offers advantages: one path may provide an avenue for benefits when the other does not, and both paths together may yield larger, accelerated benefits as compared to having a single route to impact.

Project Evaluation
The ATP, like other federal programs, is required by law to report on its performance.⁽²⁾ The ATP established its evaluation program soon after it began, even before evaluation was widely required by Congress. The Economic Assessment Office (EAO) of ATP plans and coordinates the evaluation of funded projects. It is assisted in this effort by leading university and consulting economists and others experienced in evaluation.

Performance is measured against the program’s legislated mission. Emphasis is placed on attempting to measure benefits that accrue not only to the direct award recipients, but also to a broader population, i.e., spillover benefits. This emphasis reflects the fact the public funding covers part of the costs of these projects, and, therefore, a relevant question is how the broader public benefits from the expenditure.

This report comprises one element of the EAO’s multi-element evaluation plan. The purpose of this report is to provide an interim assessment of the status of ATP-funded projects several years after they are completed. Although the ultimate success of the ATP depends on the long-run impacts of the entire portfolio of ATP projects, the performance-to-date of this partial portfolio of 50 projects provides partial answers. This study addresses the question of what the public investment of $104 million in the 50 projects has produced several years after completion of the research and what the outlook is for continued progress.

It utilizes another element of the ATP’s larger evaluation program: detailed economic case studies of selected projects. It draws from these more in-depth case studies, where they exist, to amplify the actual and prospective economic impacts of the completed projects. Other evaluation activities of the ATP include database development (i.e., a tracking of project developments through the life of the project and into the post-project period); surveys; statistical and econometric studies; model development; and special issue studies.⁽³⁾

Study Approach
From the moment that ATP funded its first group of 11 projects in the 1990 competition, program administrators, the administration, Congress, technology policymakers, industry, and others in this country and abroad were keenly interested in the outcome. But technology development and commercialization are lengthy processes, and it takes time to produce results.

Now, as the program completes its first decade of operation, there are a growing number of projects that have completed their ATP-funded research and moved into the post-project period. This group of 50 projects makes it possible to look at the projects several years after the ATP-funded research has been completed—allowing sufficient time for knowledge to be disseminated and progress to be made toward commercial goals. The larger group of projects makes it possible to form a portfolio view, compile aggregate statistics, and analyze the results in terms of their implications for overall program success.

A first step was taken toward this goal with the publication of a report in 1999 on the first 38-completed projects.⁽⁴⁾ This report takes the next step, by extending coverage to a total of 50 projects and adding consolidated performance metrics. It draws from and builds on the previous report.

At the core of this study are 50 mini-case studies covering each of the completed projects. Each of these briefly tells the project story, recounting its goals and challenges, describing the innovators and their respective roles, and assessing progress to date and the future outlook. Photographs illustrate many of the projects.

Although the particulars vary for each project, certain types of data are systematically collected for all of them. Consistent with ATP’s mission, the evaluation focuses on collecting data related to the following dimensions of performance:

• Knowledge creation and dissemination, which is assessed using the following criteria: recognition by other organizations of a project’s technical accomplishments; numbers of patents filed and granted; citations of patents by others; publications and presentations; collaborative relationships; and knowledge embodied in and disseminated through new products and processes.
• Commercialization progress, which is gauged in terms of the attraction of additional capital for continued pursuit of project goals, including resources provided by collaborative partners; entry into the market with products and services; employment changes at the small companies leading projects and other indicators of their growth; awards bestowed by other organizations for business accomplishments of project leaders; and the analyst’s assessment of future outlook for the technology based on all the other information.
  

The approach is to provide in an overview chapter the aggregate statistics of interest across the set of 50 projects, such as the total number of patents and the percentage of projects whose technologies have been commercialized. In addition, the aggregate statistics are combined to produce composite project metrics for overall performance. The composite performance scores allow one to see at a glance the robustness of a project’s progress towards its goals. Underlying the simple scores is a wealth of data.

Sources of Information
Data for the 50 projects were collected from many sources: ATP project records; telephone interviews with company representatives; interviews with ATP project managers; company websites; the U.S. Patent and Trademark Office; in-depth project studies conducted by other analysts; academic, trade and business literature; news reports; filings at the Securities and Exchange Commission; a previous study of the first 38 completed projects; and business research services, such as Dun and Bradstreet, Hoover’s Online, Industry Network, and CorpTech. Each one of the individual project write-ups was reviewed for accuracy by the project’s lead company and ATP staff.

Study Limitations and Future Directions
The 50 projects are divided into two groups: 1) the data for the first 38 projects were collected mainly between 1997 and late 1998, and 2) the data for the next 12 projects were collected during 1999 and early 2000. ⁽⁵⁾ Since developments continue to unfold for most of these projects, the output measures for the earlier set of 38 may have changed significantly since the data were collected.⁽⁶⁾ Even outputs for the later set of 12 projects may have changed since the data were collected. ⁽⁷⁾ The cases provide a snapshot of progress several years after the completion of the ATP-funded projects.

The cases, although undertaken at different calendar dates, are conducted within about the same interval of time after ATP funding ended. Yet, different points in each technology’s life cycle may be captured, depending on the technology area. Information technology projects, for example, may be expected to be further along than advanced materials and chemical projects. Examined at a later time, there may be less (or more) difference in the accomplishments among projects in different technology areas.

This study tracks outputs leading to knowledge dissemination but it does not assess the actual commercialization efforts by others who acquire the knowledge. The tracking of commercialization efforts is limited to the direct path of impact (i.e., commercialization by the award recipients or innovators).

Future studies may add mini-cases for additional completed projects to this group of 50, which would also be conducted several years after project completion. Such an extension would provide a more comprehensive view of progress made by projects at a comparable interval in time after the ATP-funded research has been completed. Additional future studies may also update these studies by looking at the projects farther out in time. An extension of the study further into the post-project period would allow for a fuller assessment of the value in the use of products and processes commercialized by the award recipients, and of the benefits resulting from the use of the knowledge (developed during the project) by others.

“Completed” and “Terminated” Projects Defined
Projects do not necessarily finish in the order funded. For one thing, they have different lengths, ranging from approximately two years to no more than five years. For another, they are required to file a final report with the ATP and have financial and other paperwork completed before project closeout. The financial closeout is done through the National Institute for Standards and Technology (NIST) Grants Office, which notifies the ATP that it considers the project completed. This study assesses the first 50 projects the Grants Office declared “completed.”

During the time the first 50 were in progress, another 16 projects were stopped short of completion. Some of these were announced as award winners but never officially started. Other projects got off the ground but were closed for various reasons with a substantial amount of the technical work still unfinished. These “terminated” projects are assessed according to the principal reasons they stopped before completion. They are treated in Appendix B.

While the terminated projects are generally regarded as unsuccessful, some produced potentially useful outputs. Appendix B illustrates a project that made an attempt to achieve its goals, only to be terminated prior to completion. It is reported in a level of detail and style similar to that provided for the 50 completed projects.

Report Organization
Chapter 1 provides a summary overview of the performance of the 50 completed projects as a group. It identifies some major outputs that appear useful as indicators of the degree of project success, and it uses these outputs in a prototype project performance rating system. A preview also notes some of the broad-based benefits that this portfolio of projects is producing and likely to produce. For additional background, the make-up of the portfolio of projects in terms of technologies, organizational structure, company size, and other features is provided.

The individual project reports presented in Chapters 2 through 6 highlight major accomplishments and the outlook for continued progress. A detailed account of the project under review is given, with attention to technical and commercial goals and achievements, information about technology diffusion, and views about the role played by ATP funding. A performance rating is assigned to each project based on a four-star scoring system. The rating depends on the accomplishments of the project in creating and disseminating new scientific and technical knowledge and in making progress toward generating commercial benefits, as well as the outlook for continued progress.

Two appendices provide supporting information. Appendix A provides a listing of technical and commercial achievements of each completed project. Appendix B provides a discussion and assessment of 16 terminated projects, together with a detailed case that illustrates how a terminated project may yield outputs of potential benefit.

____________________
1. The current edition of the kit and other program materials may be obtained on ATP’s website (www.atp.nist.gov), by e-mail (atp@nist.gov), by phone (1-800-ATP-Fund or 1-800-287-3863), or by mail (ATP, NIST, 100 Bureau Drive, Stop 4701, Gaithersburg, MD 20899-4701).

2. The Government Performance and Results Act (GPRA) is a legislative framework for requiring federal agencies to set strategic goals, measure performance, and report on the degree to which goals are met. An overview of the GPRA is provided in Appendix 1 of the General Accounting Office Executive Guide, Effectively Implementing the Government Performance and Results Act, GAO, Washington, D.C., GGD-96-118, 1996.

3. For a description of the ATP evaluation plan, see Ruegg, “Assessment of the ATP,” The Advanced Technology Program, Challenges and Opportunities, Board on Science, Technology, and Economic Policy, National Research Council, Washington, D.C., National Academy Press, 1999, pp. 71-81. Published economic studies of the ATP are available at the ATP website (see above), and can be requested by calling (301) 975-4332.

4. See William F. Long’s report, Performance of Completed Projects, Status Report Number 1, NIST SP 950-1, National Institute of Standards and Technology, Washington, D.C., March 1999.

5. William Long collected the data for the first 38. Jonathan Tucker, Chris Hansen, Josh Rosenberg, Jon Dryfus, Benjamin Fletcher, Kathleen McTigue, Michael Walsh, Mariah Tanner, and Karen Seeh collected the data for the next 12, supplemented by data collected by Rosalie Ruegg.

6. In several cases, more recent information is brought into the report to determine if the designated “top performing projects” have continued to make progress.

7. Ibid.

Return to Table of Contents or go to next section.

Date created: April 2002
Last updated: April 12, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)