 |
|
|
|
|
|
|
|
|
 |
 |
 |
 |
 |
 |
 |
 |
|
|
|
||||||||||||||||||
|
|
Measuring ATP Impact Appendix B — Significant Recent StudiesThe ATP Economic Assessment Office measures the success of the Advanced Technology Program through a variety of evaluation studies aided by leading experts. All the recent studies described in this appendix can be found at www.atp.nist.gov/eao/eao_pubs.htm. Technology Adoption Indicators Applied to the Flow-Control Machining Project. An idea from the 1995 ATP competition produced a new automobile finishing process called Flow-Control Machining (FCM), which increases the precision of cast-metal parts for interior fluid flows. This economic study develops a set of technology adoption indicators (TAIs) capable of selecting and analyzing possible spillover applications for FCM technology. When applied to engine manufacturers for lawnmowers and airplanes, TAIs revealed that the lawnmower industry is more likely to adopt this new technology, due in part to new EPA regulations, with significant projected savings in GDP. NISTIR 6888 (Economic Study). May 2003. Hayden Brown (NIST) and Mark Ehlen (Sandia National Labs). Inter-Industry Diffusion of Technology That Results from ATP Projects. This report describes the inter-industry diffusion of technology that might result if ATP-funded projects used U.S. input-output (I-O) tables to identify the fit of those projects within the U.S. industrial structure. These tables can also track the most direct path of benefits flowing to other industries. In industries represented by an ATP participant, the intensity of purchase activities by similar companies in that industry could provide a useful indicator of the likelihood of spillovers; such an indicator could be quantified by I-O tables. Rankings can be derived from 1998 I-O tables for 36 ATP projects that were funded from 1992-1996 and resulted in commercialization. NIST GCR 03-848 (Economic Study). April 2003. Joel Popkin (JPC Economic Consultants). Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data. This study uses empirical methods to evaluate the effects of participation in ATP-funded consortia on the research productivity of consortia members. The authors developed a data set for one group of firms that participated in ATP-funded research consortia, and for a second control group that were never involved. Innovative output was measured using patent data. The findings revealed a positive relationship between the firms' intensity of participation in research consortia and their overall research productivity-participation in one additional ATP-funded consortium per year would increase a firm's patenting that year by as much as 8 percent. Japanese data included in the study validated the fact that consortia have a positive impact on research productivity. NIST GCR 02-830 (Economic Study). December 2002. Mariko Sakakibara (UCLA) and Lee Branstetter (Columbia Business School). Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes. In evaluating ATP, the increased innovation of participant firms serves as an important indicator of program success. This study measures innovation outcomes by the number of patents granted and by a statistical analysis of firms before and after ATP project participation. The study showed that ATP has a positive effect on innovation in firms, and participation in the program increases firms' patenting, relative to their patenting prior to the ATP award. The study also showed that joint venture participation and university collaboration have positive impacts on innovation, as measured by increased firm patenting activity. NISTIR 6943 (Economic Study). December 2002. Lynne G. Zucker and Michael R. Darby (UCLA), and Andrew J. Wang (NIST/ATP). Universities as Research Partners. This study seeks to gain a better understanding of the performance of university-industry research partnerships by surveying a sample of pre-commercial research projects funded by ATP. Although results must be interpreted cautiously because of the small sample size, the study finds that projects with university involvement tend to be in areas involving "new" science, and therefore the projects may experience more difficulty and delay-but also are more likely to end in success. This finding implies that universities are contributing to basic research awareness and insight among the partners in ATP-funded projects; therefore universities are important to U.S. innovation. NIST GCR 02-829 (Economic Study). June 2002. Bronwyn H. Hall (UC/B), Albert N. Link (UNC/G), and John T. Scott (Dartmouth). Winning an Award from the Advanced Technology Program: Pursuing R & D Strategies in the Public Interest and Benefiting From a Halo Effect. This study addresses two questions: (1) how a firm's R & D strategy relates to the goals of ATP and affects the chances of winning an award from the program; and (2) how winning an award affects a firm's success in raising additional funds for a proposed research project. Data from a 1999 survey show that award winners are more likely to behave in ways that enhance the transfer of knowledge to-and the reception of technology by-other firms. Award-winning companies are better networked than non-winning applicants and exhibit a greater willingness to share research findings. Award-winning companies are also more likely to form partnerships to open up new innovation pathways. The study finds that award-winning firms have greater success in attracting additional funding for their ATP projects from other sources. NISTIR 6577 (Economic Study). March 2001. Maryann P. Feldman (Johns Hopkins) and Maryellen R. Kelley (NIST/ATP). Understanding Private-Sector Decision Making for Early-Stage Technology Development. This report examines trends in management of corporate R & D and how new market realities affect the ways that corporations manage and support early stage technology development (ESTD) activities. It is based on interviews with senior executives and investors from corporations across eight industry sectors and eight venture capital firms. Of $181 billion invested in R & D activities by U.S. firms in 2000, only $13 billion funded the types of ESTD activities that target delivery of radical innovations to market. Variations found across industries are shaped by forces such as the increasing sophistication required to develop new technological innovations, the increasing pressure on corporate R & D divisions to demonstrate ROI, and the importance of the lifecycle position of specific industries and individual companies. NIST GCR-02-841B (Special Issues Study). September 2003. Nicholas Demos (Booz Allen Hamilton), and Philip E. Auerswald, Lewis M. Branscomb, and Brian K. Min (Harvard University). Bridging From Project Case Study to Portfolio Analysis in a Public R & D Program: A Framework for Evaluation and Introduction to a Composite Performance Rating System. This paper presents a framework for evaluating both individual projects and a portfolio of projects in the "mid term"-3-5 years after project completion. It introduces a prototype evaluation tool, the Composite Performance Rating System (CPRS), designed for ATP but adaptable to other programs. CPRS uses uniformly collected indicator metrics to rate each of ATP's completed projects in what was added to the national scientific and technical knowledge base, the knowledge it disseminated, and the technology commercialized into new products or processes. The indicator metrics compute star ratings (0 to 4 stars) to provide a quantitative assessment of the first 50 completed projects in the ATP portfolio: 16 percent received 4 stars; 26 percent received 3 stars; 34 percent received 2 stars; and 24 percent received 0 or 1 star. NIST-GCR 03-851 (Special Issues Study). August 2003. Rosalie Ruegg (TIA Consulting, Inc.). A Toolkit for Evaluating Public R & D Investment: Models, Methods, and Findings from ATP's First Decade. This comprehensive report uses the large body of evaluation techniques and 45 selected studies developed by ATP during its first decade to provide an evaluation framework-a directory of methods, tools, techniques, principles, explanatory information, and best practices. These tools and techniques develop the body of knowledge about the behavior of participating companies, the degree of collaboration, spillover effects, interfaces with state and international technology programs, ATP's performance at large, and knowledge about evaluation itself. A cross-cutting look at study findings confirmed results from individual studies indicating that ATP is achieving its overarching objectives, leading to broadly distributed economic benefits:
NIST GCR 03-857 (Special Issues Study). July 2003. Rosalie Ruegg (TIA Consulting, Inc.) and Irwin Feller (AAAS and Pennsylvania State University). Between Invention and Innovation: An Analysis of Funding for Early-Stage Technology Development. This study addresses the distribution of funding for early-stage technology development across different institutional categories and compares government programs with private sources in terms of magnitude. The study also looks at the difficulties that firms face when attempting to find funding for early-stage, high-risk R&D projects. To arrive at a reasonable estimate of the national investment in early-stage technology development, the authors relied on the observations of practitioners that were gathered during a series of workshops held in the United States. They also collected data available on early-stage technology development investments from other studies and from public statistical sources. Findings include:
NIST GCR 02-841 (Special Issues Study). November 2002. Lewis M. Branscomb and Philip E. Auerswald (Harvard University). Different Timelines for Different Technologies: Evidence from the Advanced Technology Program. To address the variations seen in the commerciali-zation of technologies from early ATP-funded projects, this study uses data collected through ATP's Business Reporting System to analyze differences in commercialization patterns for these technologies. Variations were apparent in the timing of initial revenues, commercialization in more mature and multiple applications, and diffusion of technologies. Based on business reports from 558 participants in 299 ATP projects funded between 1993 and 1998, business expectations and strategies were examined for nearly 1,200 commercial applications. Differences in technology type-information technologies, biotechnologies, manufacturing, and electronics- are also examined within an innovation lifecycle framework to illuminate differences in diffusion patterns. NISTIR 6917 (Survey Data Results). November 2002. Jeanne Powell (NIST/ATP) and Francisco Moris (NSF). Survey of ATP Applicants 2000. To help assess the effectiveness and impact of ATP, the Economic Assessment Office sponsored a survey of all applicants in the 2000 funding competition. The resulting evaluation tool aids in assessing overall characteristics of applicants and in comparing program effects on awardees and nonawardees. All for-profit company applicants to ATP in 2000 were included in the survey sample; other organizations, such as universities and non-profit organizations, were not included. Survey responses were obtained from a total of 346 companies, including 74 companies that were awarded funding as well as 272 companies not selected for an award. Survey findings confirm the significant impact of ATP. NIST GCR 03-847 (Survey Data Results). June 2003, Westat (Rockville, MD). Determinants of Success in ATP-Funded R & D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects. This study explores the growing importance of collaborative ventures to the nation's economic strength, the factors that make them work, and the role of government in fostering collaboration. The focus is on 18 ATP-funded automotive industry joint ventures initiated between 1991 and 1997. Factors in success include trust, information sharing, an optimal number of participants, companies with complementary skills, personnel stability, cost containment, and a high level of company commitment. Findings suggest that ATP provides funding at critical stages, accelerates research, improves outcomes, and encourages partners to take on higher risk and longer-term research. ATP also helps joint ventures to overcome barriers to collaboration and helps projects run more smoothly, albeit with some loss of flexibility on the part of the companies. NIST GCR 00-803 (Case Study). December 2001. Jeffrey H. Dyer (BYU) and Benjamin C. Powell (University of Pennsylvania). Benefits and Costs of ATP Investments in Component-Based Software. From 1994 to 2000, ATP provided $42 million to support 24 projects under its focused program in Component Based Software for building large software systems by assembling readily available components. This study assesses the impact of the ATP-supported projects using quantitative and qualitative analyses. Results show that two-thirds of the funded projects achieved their technical objectives. Viewed as an investment portfolio, the 24 projects delivered social returns exceeding reasonable benchmarks for public or private investment. The authors calculate a net present value of $840 million and benefit-to-cost ratio of 10.5, suggesting that the expenditure of public funds was worthwhile. GCR 02-834 (Case Study). November 2002. William White and Michael P. Gallaher (RTI). Closed-Cycle Air Refrigeration Technology for Cross-Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries. ATP co-funded a 1995 joint venture to design, fabricate, and pilot test closed-cycle air refrigeration (CCAR), a new industrial technology that uses environmentally benign air as the working fluid. Market analyses showed the U.S. food processing industry to be a promising end market, where ultra-cold temperatures (-70°F to -150°F) help to improve food safety and reduce weight loss, dehydration from evaporation, and environmental emissions. Against a $2.1 million ATP investment and $2.2 million in corporate funds, the project has a net present value of $459-$585 million (2001 dollars), an internal rate of return of 83-90 percent, and a benefit-to-cost ratio of 220:1 to 280:1. The study concludes that CCAR technology would not have been developed without ATP funding. NIST GCR 01-819 (Case Study). December 2001. Thomas Pelsoci (Data Research Company). Low-Cost Manufacturing Process Technology for Amorphous Silicon Detectors: Applications in Digital Mammography and Radiography. This case study examines the 1995-2000 ATP-supported joint venture involving General Electric Global Research and PerkinElmer, Inc., to develop a low-cost manufacturing process for fabricating amorphous silicon detector panels used in digital mammography and digital radiography systems. The GE Medical Systems Senographe® 2000D system resulted from the ATP-funded project. This unit has proven to issue 20 percent fewer false positive results and therefore requires fewer patient recalls than conventional systems. Each unit is associated with $63,360 in medical savings per year, and the original $1.575 million ATP investment has resulted in technology estimated to be worth $219-$339 million (2002) dollars in benefits to health care industry users and patients. NIST GCR 03-844 (Case Study). February 2003. Thomas M. Pelsoci (Delta Research Company). A Study of the Management of Intellectual Property in ATP-Awarded Firms. Based on six case studies developed from interviews of ATP project participants, this paper examines the behavior of firms proposing research projects to ATP and whether such firms select research that minimizes the likelihood that other firms might benefit from resulting intellectual property. The six case studies represent two technology areas, and include single company projects and joint ventures. The findings suggest that intellectual property concerns do not affect the research that single company applicants propose but do affect a company's decision to apply as a single company applicant or joint venture. The findings also show that when firms apply as joint ventures, they may pursue strategies for maintaining control of their intellectual property so that diffusion is minimized. ATP Working Papers Series 00-01. August 2003. Julia Porter Liebeskind (University of Southern California). Catalyzing the Genomics Revolution: ATP's Tools for DNA Diagnostics Focused Program. The Human Genome Project began in 1990 as a multi-agency effort in the federal government that sought to determine the complete sequence of the DNA in the human genome by 2006. ATP participated in this effort with its Tools for DNA Diagnostics Focused Program, with competitions in 1994, 1995, and 1998; it also funded DNA tools projects in general/open competitions. Through 2002, ATP had committed more than $138 million to cooperatively fund 42 R & D projects on DNA tools. This working paper summarizes ATP's contributions to the field of DNA research, which include many innovative technologies along with the intellectual property portfolios of ATP-participating companies that have benefited an emerging industrial sector. ATP Working Papers Series 04-01. July 2004. Acknowledgments Special thanks to the ATP staff who contributed to this report: Steve Campbell and Holly Jackson for supplying data and statistics; Lee Bowes for project coordination; Connie Chang, Stephanie Shipp, and Lorel Wisniewski for their reviews, suggestions, and revisions; and ATP Acting Deputy Director Elissa Sobolewski for reviewing the final draft. Appreciation also goes to Robert Matzen, Nancy Reese, and Julie Tabaka of Akoya for their assistance with writing and layout design. We would like to thank the following ATP-participating organizations for supplying images used in this report: ColorLink, Inc., Cyclics Corporation, MicroFab Technologies, Inc., MTI Microfuel Cells, Inc., Nanophase Technologies Corporation, Osiris Therapeutics, Inc., Strongwell Corporation, and Zyvex Corporation. Contact ATP at:
Return to Table of Contents. Date created: March 10, 2005 |
 |
ATP website comments: webmaster-atp@nist.gov / Technical ATP inquiries: InfoCoord.ATP@nist.gov. NIST is an agency of the U.S. Commerce Department |
 |