 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
|
NISTIR 05–7174
Evaluation Best Practices and Results: The Advanced Technology Program 5. Examples of ATP’s StudiesATP uses a multi-faceted approach to evaluation. Reflecting the richness and diversity of studies, several examples of studies, plus highlights of findings from those studies, are presented below. These examples include policy studies, Status Reports of completed ATP-funded projects, selected survey results, and benefit-cost studies. These examples are provided to show the depth and breadth of our assessment work. DEFINING ATP’S ROLE IN THE POLICY ARENA In a report commissioned by ATP’s Economic Assessment Office, Branscomb and Auerswald (2002) address the question: how important is the public role in funding early-stage technology development? They estimate that between $5 billion (2 percent) and $36 billion (14 percent) of overall national R&D spending in 1998 was devoted to early-stage technology development. The relatively small share of total national R&D spending devoted to early-stage technology development, which arguably drives future economic growth, supports the view that there is a “funding gap” in which the amount of funding currently available is less than what is socially optimal or desirable. Of particular interest is the finding that the federal government, and not organized venture capital, is a major funding source for early-stage technology development. The federal govern-ment provided approximately 20 percent to 25 percent of funding toward early-stage technology development, with the Advanced Technology Program as one of the principal federal programs focused in this critical area. ATP targets technology development that venture capitalists do not address. Venture capitalists, state governments, and universities contribute only between 8 percent and 16 percent toward early-stage technology development. PROJECT AND PORTFOLIO ASSESSMENT Status Reports are descriptive mini–case studies for each completed ATP project, written several years after ATP funding ends. Status Reports address how well the project performed against ATP’s mission objectives. A performance rating for each project (zero to four stars) is computed using a uniform set of data. The aggregation of stars provides a portfolio view of ATP performance. For example, aggregating the performance ratings for the first 120 ATP projects shows the following distribution:
The largest group of projects, 31 percent, fell into the three-star category. These projects show strong progress. Combining these with the 10 percent rated outstanding shows an impressive 41 percent of projects performing at a high level. Twenty-nine percent fell into the two-star category. These projects show moderate progress but are not particularly robust overall. Thirty percent of the projects scored one star or less, which is not surprising, given that ATP projects are high-risk R&D and not all projects are expected to succeed. Projects may fail for technical reasons, business reasons, or a combination of both. Each status report also includes a patent tree for each patent filed during or after the ATP project ends to show the citation of the patent in subsequent patents. This is one way to portray knowledge spillovers. ANSWERING THE COUNTERFACTUAL QUESTION: WHAT HAPPENS WITHOUT ATP? The Survey of ATP Applicants was conducted in 1998, 2000, and 2002. This survey was admin-istered to all applicants in the previous competition year in order to compare the company and project characteristics of awardee and non-awardee companies soon after awards are announced. It addresses the counterfactual question, what happens when a project does not receive ATP funding. The Survey of ATP Applicants asks organizations that did not receive awards what happened to their projects once they did not receive funding from ATP. The survey results find that 41 percent of these projects are not pursued, 40 percent are pursued on a smaller scale, and of those pursued on a smaller scale, four out of five reports that the project scope is reduced to below 40 percent of the proposed ATP project. Evidence from the Survey of ATP Applicants shows that ATP is successful in directing funding to projects that have higher technical risk and longer time horizons than projects proposed by nonawardees (ATP, 2003a). A measure of technical risk is the probability that a project will not achieve its technical goals.
Proposed ATP projects for both awardees and nonawardees are higher risk and have a longer time horizon than to ‘typical’ R&D projects. ATP awardees report a greater contrast between their proposed and typical R&D projects, compared to nonawardees. A key finding is that ATP awardees attract additional funding since submitting their ATP proposal. This phenomenon is referred to as the halo effect. For example, three out of four awardees report increased funding, while one out of four non-awardees reports increased fund-ing from internal company sources. ATP awardees are also more likely to receive funding from external sources. One out of three awardees reports increased funding, and only one out of five non-awardees report increased funding from external sources. MEASURING ACCELERATION EFFECTS The Business Reporting System (BRS) allows an examination of ATP awardees from a longitu-dinal perspective and from a cross-sectional perspective. Responses to the BRS surveys indicate that ATP funding accelerated R&D in 9 out of 10 organizations. Of those organizations that indicated they were ahead in their R&D cycle:
ATP participants report that the acceleration of R&D reduces the time it will take to bring products to market or to implement new production processes. Reduction in time-to-market by two years or more is anticipated for about three out of five planned commercial applications.(1) MEASURING OUTCOMES—BENEFIT-COST STUDIES Benefit-cost studies are one of the primary ways to measure outcomes quantitatively. Outcomes are difficult to measure because one must make assumptions about the impact of the new tech-nology and acceptance by buyers. The results of individual benefit-cost studies can be aggregated to look at the impact (usually prospective estimates) across ATP. The net social benefits from about 40 ATP projects are estimated to be $18 billion. This is a more than adequate return on an investment of $2.2 billion that ATP has made between 1990 and September 2004. ATP projects produce large benefits. One example of a benefit-cost study is Low-Cost Manufacturing Technology for Amorphous Silicon Detectors, a joint venture project funded by ATP in 1995 (Pelsoci, 2003). Digital mammography and radiography systems are innovative technology solutions to the diagnostic and productivity limitations of conventional x-ray systems. The new process, which was implemented in 2004, reduces fabrication costs by approximately 25 percent without compromising performance. On the basis of 33 million mammography and 68 million chest x-rays per year, estimated benefits are $125 to $193 for every $1 that ATP has spent. Societal benefits include avoidance of unnecessary medical procedures as a result of lower false-positive rates, improved breast cancer detection, reduced patient exposure to radiation, and reduced examination time. ___________________ Go to next section. Date created: July 20,
2005 |
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov. NIST is an agency of the U.S. Commerce Department |
 |