Performance of 50 Completed ATP Projects Status Report - Number 2 NIST SP 950-2 Chapter 1 - Overview
Part 5 - Overall Project Performance

Composite Performance Scores
A variety of output data has been systematically collected for the 50 projects and presented in aggregate form in the preceding sections. The purpose was to assess the performance of the group of projects in two dimensions: 1) knowledge creation, 2) dissemination, and 3) progress toward commercialization.

In this section the focus shifts to combining the collection of outputs for each individual project, to measure how it performed overall. The result is a composite performance score for each project, computed from an extensive set of outputs linked to the ATP mission and goals. The composite performance scores are reported in chapters 2 through 6. The distribution of scores is reported here.

The composite performance scores are computed from available output data⁽²⁰⁾ indicating the creation and dissemination of knowledge, i.e., a) award-winning technologies, b) patent filings, c) publications and presentations, d) new and improved products/processes now or expected, and e) collaborations. These output measures are combined with available data indicating progress toward commercialization of the new technology, i.e., a) attraction of capital, including resources made available for commercialization through collaborative activities, b) employment gains, c) company awards for business success, d) getting product and processes into the market, and f) outlook for continued future progress by the award-recipient companies. The resulting composite performance scores are computed for each of the 50 projects. The scores are expressed in terms of a zero-to-four star rating system, where a score of one star or less signals poor overall performance; two stars, moderate performance; three stars, strong performance, and four stars, outstanding performance.⁽²¹⁾

These ratings should be viewed as roughly indicative of overall performance. Limitations include the fact that not all relevant effects are captured; alternative algorithms for computing the composite ratings may be superior to those used, and the ratings at best reflect performance in terms of the underlying project output metrics, and not in terms of the ultimate national economic benefit from the project.⁽²²⁾

Scoring the First 50 Completed Projects
Figure 1.9 shows the distribution of the 50 projects by their composite performance scores. Perhaps not surprisingly, the largest group of projects, 34 percent, scored in the two- star category—moderate progress, but not particularly robust overall. Sixteen percent (8 projects) scored in the top category, receiving four stars. Another 26 percent (13 projects) scored in the three-star category, also strong projects. Twenty-four percent of the projects scored one star or less—perhaps not surprising for companies taking on difficult goals.

Figure 1.9 Distribution of Projects by Overall Performance Score

The 8 top-scoring projects overall included 7 single-applicant projects led by small companies and 1 large joint venture led by an industry nonprofit organization. Leaders of these projects include Engineering Animation, Inc.; Integra LifeSciences; Aastrom Biosciences, Inc.; Cree, Inc.; Tissue Engineering, Inc.; Torrent Systems, Inc.; American Superconductor Corporation; and the National Center for Manufacturing Science.

The next tier of relatively strong projects includes 8 projects led by small companies, 1 project led by a large company, 1 project led by a medium-sized company, and 3 joint venture projects.

Performance by Technology Areas
Figure 1.10 illustrates the distribution of projects in the five technology areas, according to their composite performance scores. The sample sizes are too small to support the drawing of general conclusions about how projects in the different technology areas will perform. For this group of 50, several observations can be made about performance to date.

Figure 1.10 Distribution of Composite Scores by Technology Areas

The Biotechnology projects exhibit a bimodal distribution, with either outstanding/strong performance or poor performance. The Information Technology projects cluster into the outstanding and moderate performance groups; none are in the weakest categories. The Manufacturing and the Electronics/Computer Hardware/Communications projects are distributed across the spectrum of performance categories. Most of the Advanced Materials/ Chemicals projects are in the moderate performance category, and a sizable fraction is in the weak-performing group. (See “Characteristics of the Projects” earlier in this chapter for the simple distribution of projects by technology area.)

Project Performance Translated into Economic Benefits
Progress signaled by strong composite performance metrics is translating into economic benefits. Let us consider, for example, estimated benefits of projects among the top scorers.

Three of the top performing projects developed medical technologies that were evaluated by economists at the Research Triangle Institute (RTI), a consulting firm in North Carolina.⁽²³⁾ RTI economists provided early estimates of the value of a new biopolymer to repair fractures, developed by Integra LifeSciences; a system for replicating stem cells, developed by Aastrom Biosciences, Inc.;⁽²⁴⁾ and a new prosthesis material—animal-derived extracellular matrix, or ADMAT—developed by Tissue Engineering, Inc.

Biocompatible Polymers for Cartilage Repair: The RTI study estimated the medical cost savings from the project led by Integra, in terms of avoiding second surgeries to remove implants, such as pins and screws, when trouble arises. It estimated benefits at $98 million, all attributed to ATP.⁽²⁵⁾ Integra has shown continued robust progress since the RTI study was completed several years ago. At the time the project analysis for this mini-study was completed in late 1999, company employment had increased four-fold, and a recent check showed yet another big jump. Through its commercial partners, Integra’s technology is becoming embodied in surgical screws, tacks, and other fixation devices for attaching soft tissue to bone in the knee and shoulder. Patient benefits include avoidance of medical complications and further surgeries associated with utilizing existing technologies, as well as lower surgical costs.

Stem Cell Replication: Because of the difficulty of estimating the value of patient pain reduction and improved health outcomes, the RTI study also based the benefits estimates for Aastrom’s stem cell expansion technology only on the reduction in procedure costs, although it is the patient pain-avoidance and improved outcome effects that were the main expected benefit of this technology. By allowing a small amount of stem cells to be multiplied into a larger quantity, the technology reduces donor clinic visits to collect the stem cells; reduces procedure hours; is easier for medical staff to perform; reduces treatment costs; substantially reduces patient pain and negative side effects; and has been shown to result in better treatment outcomes.

RTI economists estimated that Aastrom’s replication system, once implemented, would save about $87 million (in 1997 dollars) in the costs of providing bone-marrow transplants for cancer treatment without the acceleration provided by ATP support and $134 million with the acceleration. The difference, $47 million, is the estimated additional value, in terms of cost savings only, attributed to the ATP, based on this one application area.

Aastrom, the award recipient, has since continued unabated pursuit of the commercialization of its Aastrom-Replicall™ System. In recent clinical trials, the system was used successfully to enable cancer patients—for which there were otherwise no donors—to receive stem cell transplants by expanding tiny amounts of cord blood samples that matched the patients.⁽²⁶⁾ According to the director of medical oncology at Hackensack University Medical Center, “These results suggest that we may have found a new treatment approach that will enable more patients to receive treatment for this very serious and often fatal disease.”⁽²⁷⁾ According to the American Cancer Society, 30,000 new cases of leukemia are expected in 2000 and approximately 20,000 people will die from the disease this year, making new, more effective treatments of great value to society.⁽²⁸⁾

Biomaterials for Prostheses: Focusing on the first expected application of Tissue Engineering’s ADMAT technology, namely, the repair of damaged knee ligaments (specifically, anterior cruciate ligaments, or ACLs), RTI economists also estimated benefits. But in this case, unlike the previous two, RTI was able to estimate benefits expected to result from improvements in the quality of life for patients receiving the treatment, by using a “quality-adjusted-life-years” index value.⁽²⁹⁾ RTI estimated about $15 billion in expected net benefits from the new technology attributable to ATP funding.

Printed Wiring Boards: Another of the top performing projects is a joint venture led by the National Center for Manufacturing Sciences (NCMS) to develop a suite of advanced technologies for producing printed wiring boards, the backbones of electronics products. Two studies conducted by Professor Albert Link of the University of North Carolina–Greensboro assessed the impact of the project’s extensive use of collaborative effort.⁽³⁰⁾ The Link studies estimate that the project’s collaborative effort produced at least a 53 percent reduction in overall research costs, resulting in an R&D savings of at least $35.5 million to produce the new capabilities needed by the industry for international competitiveness.

Evidence is strong that the project produced leap-frog technologies that have yielded productivity improvements for member companies and improved the competitive position in the world market for PWBs of the hundreds of small U.S. suppliers who were themselves lacking in advanced R&D capability.⁽³¹⁾ The award-winning papers, new products, and other knowledge dissemination activities by the joint venture have helped to spread the new capabilities across the entire industry.

At a recent technology exposition by ATP-funded companies, an advanced circuit board was displayed that incorporated many of the innovations developed by the ATP-funded project. A small U.S. company that did not participate directly in the project was said to have produced it. This new PWB was provided by NCMS as evidence of the effective knowledge flow from the project to others.⁽³²⁾

Scalable Parallel Programming: Another of the top performing projects is a project led by Torrent Systems, Inc. The project developed a component software system that insulates programmers from the complexities of parallel programming while allowing them to use it productively in scalable applications. Torrent delivered this new capability in its software product, OrchestrateTM. An early company user of the new software reportedly was able to increase its revenue by $100 million per year.⁽³³⁾ Torrent’s technology is making it possible for eBusinesses and other companies to process and analyze unlimited volumes of data. Torrent was listed in Computerworld’s “100 Hot Emerging Companies” in 1998, in addition to receiving a number of other awards recognizing both its software technology and business acumen. Although Torrent had fewer knowledge-dissemination outputs than the other top performing projects, its exceptional showing on the commercialization side boosted it into the four-star group.

HTS Wire: The project led by American Superconductor Corporation (ASC) was another of the top projects in terms of performance. The new technical capability developed by this company is enabling it to produce high-temperature wire for use by electric utilities and as a component in motors, transformers, and specialty magnets to reduce their energy consumption. With an estimated sales volume of $15 million in 2000, and a rapid sales growth rate, this small company is launching the commercialization of its technology. The technology’s ability to reduce energy costs has taken on increased significance in the face of rising energy prices.

Visualization Software: As in the preceding examples, Engineering Animation, Inc., leader of another of the top performing projects, has aggressively and successfully pursued applications of its award-winning imaging software capabilities developed in the ATP-funded project. The company used its ATP-funded technology to improve the training of doctors, as well as to guide medical procedures. Patients in a particular surgical procedure that employed the visualization software reportedly had better outcomes as a result.

Founded by two professors and two graduate students in 1990, the company had 20 employees at the time ATP made the award to it. The company now employs approximately 1,000, had sales of $71 million in 1999, and experienced a sales growth rate over the past year of 34 percent. According to company officials, the ATP award allowed it to significantly extend its capabilities in computer visualization and computations dynamics and to form important collaborative relationships that it has since been able to leverage in many different directions. Recently, it has extended and deployed its award-winning visualization capabilities to develop a virtual factory technology, implemented recently at Ford, which enables faster design and analysis of factory models. Its many customers and clients have benefited from the company’s extended capabilities.

To these examples from the group of four-star projects, we can add examples of other strong projects from among the three-star projects that are delivering important economic benefits. Some of these projects in this group may outperform the four-star projects in some ways, but received lower composite scores based on all the recorded outputs.

Auto Body Manufacturing Assembly: One such project, led by the Auto Body Consortium (ABC), has generated documented production cost savings and improved automobile quality, as well as the potential for extending these same kinds of benefits to the manufacturing assembly of other products. A study by the CONSAD Research Corporation projected economy-wide benefits of about $3 billion in 2000, attributable to deployment of the ABC technology in automobile production.⁽³⁴⁾ Efforts are underway to extend the use the technology from the automobile industry to other industry sections.⁽³⁵⁾

DNA Sequencing System: Another example from the three-star group is provided by a project led by Amersham Pharmacia Biotech, which is credited with accelerating development of an enzyme important to the speed of the human genome project—where timing is of enormous significance.

Highly Integrated Digital Circuits: An example of a particularly strong commercialization showing from the three-star group is a project led by Vitesse Semiconductor Corporation. Vitesse has successfully applied gallium arsenide (GaAs) material in the volume production of highly integrated and very complex digital circuits. Most of the world’s telecommunications companies now use Vitesse chips; virtually every long-distance call passes through its integrated circuits. The project’s main avenue for knowledge dissemination was through commercialization, rather than patents and publications.

To these examples, we can add a number of other promising technologies—technologies that may improve productivity, facilitate better weather forecasts, improve communications, enable new drug discovery, reduce energy costs, and lower loss of limb and life globally by improving detection of old land mines and toxins.

What Difference Did ATP Make?
The ATP aims to fund projects that would not take place in the same timeframe, scale, or with the same goals without ATP’s support. A project may be successful in terms of achieving its goals, but if the same accomplishments would have occurred in the same timeframe without the ATP, then the program has not had the intended effect. Similarly, evaluation studies of ATP should apply the principle of “additionality” to correctly distinguish between benefits that would likely have occurred anyway and those benefits that are reasonably attributable to the ATP.

In preparing the 50 individual mini-case studies in this report, project leaders were questioned about the role ATP funding played in their projects.⁽³⁶⁾ Their answers are presented in the detailed discussions of Chapters 2–6 and summarized in Table 5.

Twenty-six of the 44 project respondents (59 percent)⁽³⁷⁾ indicated that they would not have done the project at all without ATP funding. Indeed, some participants said their companies would have gone out of business had the ATP award not been made. ⁽³⁸⁾

Eighteen respondents (41 percent) said they would have attempted the project at some later date or slower pace, and that the ATP funding enabled them to accelerate the technology development. Thus, for the 44 projects responding—none would have been completed in the same time frame without ATP funding.

The acceleration of some of the projects may seem short, but the value of even a small acceleration can be substantial. The six-month lead in developing Thermo-Sequenase, a DNA polymerase that is both thermostable and accurate, for DNA sequencing is an example of how a small acceleration may be significant. Speed in developing and commercializing a technology can also mean global market share for U.S. producers.

Receipt of an ATP award also has reportedly enhanced the ability of some companies to raise additional capital and form collaborative relationships for research and commercial activities. Several reported that receipt of the ATP award had enabled them to gain in international competitiveness.

Table 5. Effect of ATP Funding on Conducting Projects
Effect on Project		Number of Projects	Percentages of Respondents
Would not have proceeded without ATP funding		26	59%
Would have proceeded without ATP funding, but with a delay (39) of:		18 41%
	6 months	1
	18 months	4
	21 months	3
	24 months	5 (40)
	5 years or more	4
	unspecified	1
Total		44

What Constitutes Success and Failure for ATP?
Because individual project failure must be allowed and tolerated in a program that focuses on overcoming challenging technical barriers to innovation, it is essential to take a project-portfolio approach to assessing ATP. And, success should be assessed against the legislated mission of the program.

Three general tests, and several additional specific tests—all derived from ATP’s mission—if applied after sufficient passage of time, should reveal the extent to which ATP has successfully met its mission: Test 1: Has the portfolio of ATP-funded projects overall produced large net benefits (i.e., benefits minus costs) for the nation? Test 2: Has a substantial share of the net national benefits accrued to citizens and organizations beyond the ATP-award recipients? Test 3: Did ATP make a substantial positive difference in the size and timing of the benefits?⁽⁴¹⁾

Additional specific tests of success include the following: Did the projects produce new scientific and technical knowledge? Did ATP increase collaboration? Were small businesses able to participate? Were manufacturing capabilities improved? Did U.S. businesses become better able to compete in global markets?

Partial Answers
While the ultimate answers to these success “test questions” depend on the long-run impacts of the portfolio of ATP projects, the performance-to-date of the subportfolio of 50 projects provides partial answers.

The performance ratings show that the majority of the projects are still alive, in the sense that progress continues to be made. More important, they reveal a core group of highly active and productive projects that are successfully accomplishing their big project goals.

The ATP awarded a total of $104.0 million to the 50 completed projects and another $10.5 million to 16 terminated projects (see Appendix B), bringing total ATP spending on 66 projects completed or terminated by May 1998 to $114.5 million. What is the public investment producing in the way of benefits?

Estimated benefits attributed to ATP from just a few of the top-performing 50 projects not only greatly exceed ATP’s funding for all of the 66 projects, they also far exceed the total of ATP costs for all of the 522 projects funded to date.

In addition to benefits exceeding costs, there is strong evidence that benefits are extending well beyond those captured by the award recipients. There is substantial evidence that knowledge was generated by the projects and that it is being disseminated to others through publications, presentations, patents, products, and other means. The patent trees developed for these projects reveal rich citing of the patents by others. The products and processes generated by the projects are also yielding benefits to others: patients are receiving spillover benefits from better medical treatments at lower costs; consumers are receiving spillover benefits when they buy superior products for which they pay less than the full value; and companies are receiving spillovers when they increase their productivity or achieve greater value added by using ATP-funded technologies.

This completes the portfolio view of the ATP. Now let us look at the 50 individual projects.

____________________
20. The available data do not capture all developments that may be important to ultimate project performance. For example, the data do not capture informal discussions and mobility of workers that may be important avenues for the flow of knowledge. They do not capture details about sales volume and the value of commercialized products and processes. Inclusion of additional effects might produce improved measures of overall performance. Also, changes in the way in which the various output measures are assigned points and combined to compute the composite score may offer potential for improved measures. The scoring system was developed by Rosalie Ruegg, TIA Consulting.

21. The performance metrics are consistent with the view of varying degrees of success—with knowledge creation and dissemination constituting partial success, and a continuation into commercialization constituting a fuller degree of success in terms of project progress. Some companies carried out their proposed research with a degree of success during the time of ATP funding, but then did not continue pursuit of their project’s larger goals after the ATP funding ended. At this stage of evaluation, ATP considers such projects only partial successes, because the direct path for achieving project goals is truncated. Such projects are not among the higher scorers in this report. The reader is reminded, however, that commercial developments along the indirect path may nevertheless occur—particularly if the project produced effective knowledge transmitters, such as patents and publications—and future performance scores may need to be revised to reflect commercial developments along the indirect path of project impact.

22. The degree of correlation between a project’s performance score and its long-run societal benefits is impossible to know at this time. On the one hand, few outputs and low scores for a project cast doubt on the likelihood that it will attain the large benefits originally envisioned. On the other, it is easy to imagine potential exceptions. For example, a single publication or patent could have enormous enabling influence on other U.S. companies who respond in a timely way with commercial activity. Or, a company may work in secrecy for a long period of time with no visible outputs, and then suddenly explode on the scene with a single output that will yield large societal benefits. Similarly, it should be noted that projects with the same scores are not necessarily equal in their potential benefits. They are, however, somewhat comparable in terms of the robustness of their progress to date.

23. RTI, A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, Prepared for the Advanced Technology Program, 1998.

24. Aastrom Biosciences, Inc., utilized human hematopoetic stem cells derived from blood or bone marrow for its ATP supported research and continues to use bone marrow and umbilical cord blood in its development efforts. The bone marrow and cord blood utilized by Aastrom during its ATP supported research complied with the regulations for the protection of human subjects as codified by the Department of Commerce at 15 CFR Part 27.

25. The quantitative estimate, which was net of costs, did not include benefits of patient health effects and pain avoided.

26. Cord blood, the blood in the umbilical cord and placenta after the birth of a child, has been found to offer an alternative source of stem cells for transplantation that allows a higher degree of tissue mismatching while still providing an acceptable transplant. As a result, a cord-blood banking infrastructure is being established, but the small amount of cells available from this source restricts its use. In recent clinical studies, several adult cancer patients needed very high-dose chemotherapy to be followed by a stem cell transplant, but no suitable bone marrow donors could be found. In the cord blood bank, small samples were found that provided a match, but the amounts were too small to be used. In the clinical trials, the Aastrom-Replicall™ System was used to replicate the number of matching cord blood cells, allowing the patients (for which there were otherwise no donors)
to receive stem cell transplants.

27. PRNewswire, April 19, 2000, Report on results from Hackensack/Aastrom Studies.

28. Ibid.

29. For a description of the use of Quality Adjusted Life Years (QALY) in evaluating patient benefits in evaluation studies, see Andrew Wang, “Key Concepts in Evaluating Outcomes of ATP Funding of Medical Technologies,” Journal of Technology Transfer, Vol. 23 (2), 1998, pp. 61-66.

30. See A. N. Link, Advanced Technology Program Case Study: Early Stage Impacts of the Printed Wiring Board Joint Venture, Assessed at Project End, Report prepared for the Advanced Technology Program, 1997.

31. The president of NCMS credited the ATP project with saving the PWB industry in the United States with its approximately 200,000 jobs.

32. ATP Technology Showcase, “Ten Years of Innovation and Impact,” Russell Senate Building, Caucus Room, April 5, 2000. Discussions with an NCMS spokesperson.

33. Information from Hoover’s on-line company search and Torrent’s website, current August 31, 2000.

34. CONSAD Research Corporation, Advanced Technology Program Case Study: The Development of Advanced Technologies and Systems for Controlling Dimensional Variation in Automobile Body Manufacturing, National Institute of Standards and Technology, NIST GCR 97-709, 1997. A new study is currently underway, led by MIT researchers, to extend and update the analysis of this project. Contact ATP’s Economic Assessment Office for further information.

35. ATP has been exploring the diffusion of technologies developed with ATP support through the Manufacturing Extension Program (MEP) that is also operated by NIST, with the ABC project as a case in point. The output measures used in computing the composite scores did not register this type of dissemination activity.

36. Throughout the project selection process, beginning with the application, ATP presses the questions of why the project requires ATP funding in order to be undertaken, what will happen if the ATP funding is not provided, and how will the expected outcome differ with and without ATP involvement. During the evaluation process, the question is again asked in a retrospective way, i.e., what happened that was different as a result of the ATP? In either case, the question is hypothetical, and the accuracy of results is difficult to establish. A recent study by Feldman and Kelley sheds light on the question of additionality. It investigated what happened in the aftermath to projects proposed in ATP’s 1998 competition that were and were not funded. The findings provide evidence that ATP made a difference to project outcomes. (See Maryann P. Feldman and Maryellen R. Kelly, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting From a Halo Effect, NISTIR 6577 (Gaithersburg, Md.: NIST, 2001).

37. Personnel changes, severe company financial distress, or lack of clarity in responses to interview questions made it impossible to include 7 of the 50 projects in this tabulation.

38. W. Long, Performance of Completed Projects, number 1, March 1999, p. 132, endnote 15.

39. Another factor potentially influenced by ATP funding—the scope and scale of the project—was not explicitly covered.

40. The Printed Wiring Board Joint Venture project had a split response: half the tasks would not have been done at all and half would have been delayed by at least a year. This result is recorded conservatively in Table 5 as a two-year delay.

41. The tests of success are taken from a presentation by Rosalie Ruegg before the National Grants Management Association’s Annual Conference, Federal Bar Association Panel, April 4, 2000.

Return to Table of Contents or go to next section.

Date created: April 2002
Last updated: April 12, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)