A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 9: A Crosscutting Look at Study Findings

This chapter rotates the analytical perspective found in Part II to focus on the findings of the evaluation studies supported by ATP. It highlights the general principle articulated in Chapter 2: that taking multiple analytical and empirical perspectives and evaluation approaches to complex questions contribute to more reliable, credible, and acceptable findings.

This crosscutting analysis of ATP’s evaluation studies is organized around the following major themes: (1) firm/industry effects, (2) collaboration effects, (3) spillover effects, (4) comparisons and interfaces with other programs, and (5) measures of overall ATP performance, including portfolio analysis, social returns, and impacts on competitiveness. A table at the beginning of each thematic section presents a listing of reports from which the findings are drawn and provides a further breakdown by sub-themes treated in the section.

Direct Private Firm Effects

Private firms play a central role in ATP’s operations. As Spender put it, “ATP harnesses private firms’ resources to the public interest without seeking to outguess market forces.” ²⁹⁴ The term “public-private partnership” further recognizes that the program works only in conjunction with private-sector partners. ATP is designed to attract private firms as partners and to influence their behavior, even as it relies on the special knowledge that firms have of their relevant markets, and on their profit motive and their sharing of the costs to increase program efficiency. It is not surprising, then, that a number of evaluation studies have provided insights as to how and why firms engage with ATP to develop new technologies, how they are affected by that engagement, and the significance of the interactions to the firms. This section summarizes related findings on private firm effects drawn from 13 studies listed in Table 9–1. Besides author and publication date, the table’s column headings indicate the six major sub-themes to be covered. ²⁹⁵

Addressing a “Financing Gap”

Private firms face barriers to innovation that have a long development and commercialization time horizon. The Harvard-MIT study sought to develop a better understanding of “the decision-making process within firms, and within outside financing sources, as it relates to the funding of early-stage, high-risk technology projects...” ²⁹⁶   It took as a point of departure a key premise leading to the establishment of ATP:

There was evidence that firms were systematically under investing in leading-edge technologies and failing to commercialize the products of their own research activities effectively. These concerns, buttressed by academic arguments pointing to a potential market failure in the area of early-stage technological developments, motivated new proposals for the role of government in the innovation system. (p. 1.)

The Harvard-MIT study looked to venture capitalists and others to help define the financing conditions faced by innovative firms. The views expressed by study participants “make clear the existence of a serious gap between the public resources available for academic and national laboratory research and the ability of private venture investors to finance research to reduce the new technical ideas to commercial form...the ‘Valley of Death’ in R&D...” ²⁹⁷

Table 9–1. Studies Informing the Impacts of ATP on Private Firms

According to David Morgenthaler, a past chair of the National Venture Capital Association and a participant in the Harvard-MIT study, “A useful way of looking at technical risk is to assess at what stage of development of a technology an institutional venture capitalist should invest.” He provided as guidance the list below. ²⁹⁸ In his view, there is “little or no role for government in the later stages of development” but in the earlier stages of R&D, where venture capitalists generally do not want to invest, private firms need support by the government. Mogenthaler’s list of why venture capitalists should and should not invest helped to define where government support is needed: Venture capitalists should:

• ... never invest to discover new scientific phenomena
• ... almost never invest to prove the scientific principle
• ... rarely invest to develop an enabling technology
• ... often invest to use a new technology to develop a product
• ... very often invest to revise and improve a product
• ... very often invest to produce a later-generation product
• ... very often invest to broaden a product line
• ... very often invest to apply a product to another application (p. 106)

Morganthaler elaborated as follows:

...it does seem that early stage help by the government in developing platform technologies and financing scientific discoveries is directed exactly at the areas where institutional venture capitalists cannot and will not go. In the analogy of the horse race, the role of the government can be to improve the bloodlines of the horses and give them some preliminary schooling. (pp. 107–108)

In a related study conducted by Harvard University researchers, ²⁹⁹ it was found that:

...most funding for technology development in the phase between invention and innovation comes from individual private equity “angel” investors, corporations, and the federal government—not venture capitalists. (p. 3)

And, further,

...the federal role in early-stage technology development is far more significant than may be suggested by aggregate R&D statistics. In general, … federal technology development funds complement, rather than substitute for, private funds. (p. 10)

The struggle of small private firms to find funding for the innovations they eventually brought to ATP has been documented in case studies. Gompers and Lerner interviewed seven small, innovative companies in the Boston area to find out what role “ATP funding has played in the company’s evolution.” ³⁰⁰ They found reasons each of the companies turned to ATP for funding, and concluded, “The Advanced Technology Program has substantially expanded and enhanced the R&D activities of our seven-company sample.” ³⁰¹ They concluded that despite the growth in venture capital funds during the mid-1990s, less than one tenth of one percent of business startups annually received venture financing. ³⁰²

Servo used the experiences of several firms to describe how ATP funding fit within the landscape of potential funding sources. As recounted by one firm spokesperson in her study, “We sought ATP funding as a means to develop novel process technology which exceeded our existing financial capability and which we were unable to convince venture capitalists and angel investors to back due to high technical risk.” ³⁰³

ATP’s Status Reports also documented circumstances in which firms were unable to obtain private sector financing, and sought ATP assistance in order to accomplish their innovation. For example, in documenting the ability of the two young proprietors of Calimetrics, Inc., to find financing for their pit depth modulation (PDM) technology, the Status Report gave the following account: ³⁰⁴

When first starting out, O’Neill and Wong had confidence in PDM’s technological potential, but the fledgling company did not have the means for sustained research. With a kick-start of $25,000 from family and friends, the company sought funding from venture capitalists and other private sources. All, however, rejected investment in PDM development, deeming the technical and market risk too high. The company faced the classic Catch-22 situation of innovation financing: funding was not available because the technology was unproven, yet it could only be proven if substantial financial resources were brought to bear on the problem. (p. 108)

Detailed case studies also shed light on the financing gap. According to Ehlen, who investigated the joint venture to develop flow-control machining technology, the end users of the technology tend not to fund the research of their suppliers and are unlikely to adopt a new process not yet proven to work; the suppliers lack the capital to do extensive in-house research—“particularly not high-risk research”; and university researchers lack the interest and resources to self-fund such a research program on their own. But with ATP funding, a supply-chain collaboration was formed to undertake the project. ³⁰⁵

Why do companies turn to ATP for funding? Several company spokespeople appearing at the National Research Council’s (NRC) workshops on ATP offered their own explanations. For example, David Ayares of PPL Therapeutics, Inc., a company working on xeno-organ transplantation, said: ³⁰⁶

To raise money, we aggressively pursued deals with several large pharmaceutical companies and with other xenograft companies. We wrote a business plan, which we showed to venture capitalists and private investors. We also pursued joint ventures. None of these efforts bore fruit. Risks and costs were too high, and payoffs too distant for potential investors. (p. 147)

Feldman and Kelley, in their survey study of a large group of 1998 ATP applicants, found that most companies turned down by ATP did not proceed with their R&D projects. If funding were readily available from other sources, it would seem that more of the companies would have been able to pursue their projects without ATP. ³⁰⁷ And, according to Powell and Lellock in their analysis of ATP Business Reporting System (BRS) data, ATP award recipients increased their own investments in their ATP-funded project as a result of obtaining ATP. ³⁰⁸ These findings provide evidence that ATP funding is complementary to, not a substitute for, private sources of R&D funds; that is, that it is filling a funding gap.

Influence on R&D Investment Choices

While some of the evaluation studies focused on the availability of funds and the timing of investment, others investigated the effects of ATP funding on firms’ R&D investment choices. Identified effects included: (1) increased innovativeness of research programs, (2) willingness to commit to longer-term research programs, and (3) ability to tackle larger scale and broader scope problems and opportunities.

Powell and Lellock, in their analysis of BRS data, found evidence of increased innovativeness attributed by firms to ATP. Figure 9–1, for example, indicates, “With the assistance of ATP funding, organizations are pursuing different R&D than they would have undertaken without ATP funding.” ³⁰⁹ Powell and Lellock found that, relative to what the project participants thought would have happened without ATP, 73% of participants in ATP projects “were more willing to accept technical risk” while 67% had “increased interest in performing longterm research” and 61% “had increased the R&D scope.” ³¹⁰

Figure 9–1. Change in the Nature of Industry R&D

Source: Business Progress Reports from 403 organizations in 198 ATP projects funded 1993–1997 — after one or more years of ATP funding.

BRS data also provided specific examples of estimates by project participants of ATP’s effect on their innovativeness. The participants identified the single most important parameter associated with technical success, and then specified the baseline metric, their project goal metric, and the expected value to be attained without ATP assistance, all in terms of the most critical parameter. This approach allowed comparisons. Table 9–2 provides a few examples drawn by Powell and Lellock from the BRS database. One of the examples is for gene sequencing. At the beginning of a project to develop advanced gene sequencing technology, the rate of sequencing was one gene per day. With ATP funding, the goal was to develop the ability to sequence 100 genes per day. Without the ATP award, the participants projected that they could only achieve a sequencing rate of five genes per day within the planned time frame. ³¹¹

Table 9–2. Illustrative Metrics Showing ATP’s Impact on Firms’ Innovation Goals

Source: Drawn from Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, p. 11.

Feldman and Kelley provided additional evidence that ATP has influenced firms’ R&D investment choices. In their comparison of winners and non-winners, they found evidence that ATP encourages firms to pursue new technical areas, outside the scope of the firm’s past R&D activities. They found that among all applicants, 28% proposed in a technical area new to the company, but of award winners, 47% proposed in a new technical area. They concluded “these differences suggest that the ATP’s cost-sharing partnership with industry is indeed underwriting the efforts of firms...to initiate risky projects in new technical areas.” ³¹²

Halo Effect from ATP Award

As indicated earlier in the report, the existence of a “halo effect” on firms receiving ATP funding has been a recurring theme among project participants. In ATP’s first survey, Solomon Associates found 100% of single-firm award recipients and 60% of joint venture participants reporting that their ATP award had “enhanced the organization’s credibility in some way which resulted in impacts ranging from enhancing the image of the company with potential investors to increased public awareness of the technology.” ³¹³ This was mentioned by many of the respondents as “the single most important effect that the ATP award has  had on your organization thus far....” ³¹⁴

In ATP’s second major survey, which focused on participants in the first three ATP competitions, Silber & Associates replaced halo terminology with “increased credibility.” It concluded that 90% of participants benefited to a “great” or “moderate extent” from enhanced credibility associated with the award. ³¹⁵ According to Silber’s report, “Credibility is almost equal in importance to the direct financial impact of the award, particularly since increased credibility can mean increased business activity.” ³¹⁶

The on-going BRS survey of ATP project participants used the terms “increased credibility” and “halo effect” interchangeably. Powell and Lellock’s analysis of BRS survey data showed 93% of participants citing increased credibility due to their ATP award. ³¹⁷ According to the researchers, “The ‘halo effect’ may be expected to be of particular benefit to ATP-funded small businesses, which have little if any market presence and typically very limited financial resources at the time of the ATP award.” ³¹⁸

Among five major ATP effects Laidlaw identified as helping project participants, was a halo effect. ³¹⁹ Laidlaw’s reporting of interviewee comments suggested the connection between the ability of firms to carry out high-risk research, as discussed in the previous section, and the halo effect:

We didn’t round up private funding until after we got ATP funding. It’s a high-risk project, and we had previously operated by bootstrapping, which would not have lasted long. (Laidlaw, quoting an interviewee, p. 30)

Feldman and Kelley’s survey of 1998 winners and non-winners rigorously tested the hypothesis that ATP confers a halo effect. ³²⁰ To do this, they took into account other factors that could have explained improvement in the firms’ ability to attract funding from other sources. As stated in the NRC’s condensed version of the Feldman and Kelley study, ³²¹

Our analysis concludes that winning an ATP award significantly increases the firm’s success in attracting additional funds from other sources for R&D activities. Our findings provide strong evidence that the ATP award confers a halo effect on winners that makes them more likely to attract other funding when compared to non-winners of the same size, and age with projects of similar business and technical quality. Thus, our results confirm that the ATP award appears to send a market signal that certifies that the firm and the technology are promising. (p. 207.)

Acceleration of Technology

Because helping U.S. businesses accelerate development and commercialization of technology is a legislated mandate for ATP, a number of evaluation studies have investigated and measured this effect. The term “acceleration,” as used here, encompasses a wide scope of changes in timing, ranging from months to years, to cases where the technology would not have been developed at all without ATP. And, as noted by Laidlaw:

...it appears that the ATP is helping to overcome two types of economic efficiency problems related to speed to market: 1. the difficulty of obtaining funding at all to undertake long-run, high-risk, enabling technology development; and 2. the difficulty of implementing coordinated, collaborative R&D management practices needed to speed the conduct of R&D and the commercialization of the resulting technology. (p. iii)

Surveys of project participants provided the earliest evidence that participating in ATP was allowing firms to speed up technology development and commercialization. In the first ATP survey, Solomon Associates found that 69% of participants had saved time in developing their technology—usually in the one to five year range—and 35% of these considered the time saved of critical importance. ³²² Three years later, Silber & Associates found 95% of participants reporting a shortened R&D cycle because of ATP, with most reducing their R&D cycle by at least two years. ³²³ ATP’s BRS, which has regularly tracked acceleration, added questions about the participants’ window of opportunity to questions about the amount of time saved and the importance of saving time. In a further departure from the earlier surveys, the BRS looked at acceleration in terms of the various applications of the technology, making it not strictly comparable to the Solomon and Silber results.

Figure 9–2, reproduced from a BRS report, ³²⁴ shows the three aspects of timing tracked by this survey: (1) the estimated reduction in time to market, (2) importance of speed, and, to put time into perspective, (3) the perceived length of the window of opportunity. As Powell and Lellock observed:

...nearly all the companies expect some reduction in the time it will take to complete the R&D phase and bring their products to market/or implement new production processes as a result of ATP funding... A reduction of at least two years is anticipated for 65% of all applications... The importance of speed-to-market is considered ‘important’ or ‘critical’ for 98% of applications... Further emphasizing the importance of acceleration, the window of opportunity for 73% of the applications to enter the marketplace is considered to be within two years after ATP funding ends; i.e., it appears that companies believe they would miss the opportunity, or a significant part of it, without the acceleration enabled by ATP funding. (pp. 11–12)

Figure 9–2. Importance of Timing

Source: Business Progress Reports for 974 applications being pursued by 464 companies in 258

The Feldman-Kelley survey provided additional evidence of an acceleration effect of ATP on R&D. As shown by Table 9–3, winners proceeded with their proposed R&D, while less than two-fifths of the study’s control group of non-ATP-winners had started any aspect of the R&D project they proposed to ATP a year after being unsuccessful in the ATP competition. ³²⁵ The results implied that the winners also might not have started without ATP, but otherwise provided no specific estimates of time saved.

Table 9–3. The Extent to Which Non-Winners Pursue the Proposed R&D Project Without ATP Funding

*Companies reporting more than one type of alliance are included twice.

Source: ATP’s Business Reporting System. Data for 747 applications being pursued by 356 companies in 198 ATP projects funded 1993–1997—after one or more years of ATP funding.

Laidlaw’s interview-based analysis of the acceleration effect produced a clearer understanding of how and why ATP caused acceleration to take place, and even placed rough values on it. In her interviews with ATP award recipients, she asked why saving time is important to the firms. She investigated how the firms actually saved time through participating in ATP and whether the effects extended throughout the firm, beyond the ATP-funded project. ³²⁶

Laidlaw found that 96% of interviewees assigned a rating of “very important” to reducing cycle time, in order to meet competition. ³²⁷ When asked how much time they had saved due to ATP participation, the median response was “by 50% or three years.” ³²⁸ Table 9–4, from the study, represents Laidlaw’s attempt to obtain quantitative estimates of the dollar value of a year of time saved. For the 54% of surveyed firms responding with quantitative estimates, “The median estimate of the economic value of reducing the applied research cycle time by just one year is $5 million to $6 million.” ³²⁹

Table 9–4. Estimates of Economic Value of a One-Year Reduction in Applied Research Cycle Time, in Order of Decreasing Value

Firms interviewed by Laidlaw identified five principal factors that helped them cut time. These are summarized in Table 9–5. The most frequently mentioned factor was “ATP’s requirement for an integrated business and R&D plan with its emphasis on concurrent engineering…” ³³⁰ Spokespeople mentioned the integrated planning requirement more often than ATP’s funding as an important factor in accelerating their technology development. Elaborating, firms spoke of the positive effect on their timing of having a well-laid-out plan and following the plan without interruption, the stability lent by ATP’s involvement, early initiation of work with potential customers and users, integrating the voice of the customer, using a systematic approach, developing definable timelines, benchmarking, and selecting potential technology applications early in the process, and enhancing documentation procedures.

Table 9–5. ATP Effects that Helped Interviewees to Reduce Cycle Time

But, if ATP’s funding effect is defined more broadly—taking into account both ATP’s direct funding plus the improved ability of firms to attract additional financial assistance from others through ATP’s “halo effect”—funding is the effect most frequently mentioned by firms as speeding their technology projects. ³³¹

Laidlaw found that 86% of those interviewed expected the time saved in the R&D stage to flow through to later project stages. ³³² Thus, speeding R&D typically speeded commercialization.

She found that 86% of firms found a carry-over of time improvements to their other technology development projects. Table 9–6 summarizes the responses, identifying ways the time reductions were transferred to other projects.

Table 9–6. Carryover of Cycle-Time Improvements to Other Projects

Source: Adapted from Laidlaw, Acceleration of Technology Development by the Advanced Technology Program, 1997, Table 10.

Developing enabling, generic technology—”technology platforms”—reportedly was a major factor allowing firms to speed other projects involving multiple applications of the ATP-funded technology. Applying the ATP-required integrated R&D and business planning to other projects reportedly also helped firms speed other projects. The ability to apply methodologies and processes developed in ATP projects to other projects was another factor identified as helping firms speed their other projects. Some firms further reported that they had developed a company culture emphasizing speed as a result of their ATP project participation. ³³³

Project case studies provided another look at ATP-induced acceleration of technology  development. The RTI case studies, for example, estimated project acceleration ranging from “one to at least 10 years” for the seven tissue engineering projects included in the study. ³³⁴ In a more complicated case, the firms in the ATP-funded Printed Wiring Board joint venture reportedly would have delayed about half their research by at least one year without ATP’s assistance, and would not have undertaken the other half at all. ³³⁵

The set of mini-case studies for the completed projects provided yet another look at acceleration. Table 9–7 summarizes the acceleration effects of ATP funding for 44 of the first 50 completed projects, those for which data were available. ³³⁶ The majority of participants in completed projects reported they would not have proceeded at all without ATP funding. ³³⁷

Table 9–7. Effect of ATP Funding on Completed Projects

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status  Report 2, 2001, p. 24.

Increased Firm Productivity

ATP’s evaluation investigated program-induced changes in patenting by award recipients to gauge program impact on firm research productivity. Two of the evaluation studies examined—the Darby et al., study and the Sakakibara- Branstetter study—provided statistical evidence about how participation in ATP affected patenting behavior. ³³⁸

In determining how to best measure the impact of ATP participation, Darby et al., identified patents as “arguably the single best measure of commercial capture of  advanced technology.” ³³⁹ They found this to be an appropriate measure because a “major purpose of ATP is to increase commercial capture of advanced technology.” ³⁴⁰ They produced an adjusted patent count measure by deflating the patent count of ATP participants using a factor for year-to-year changes in the average rate of patenting for all U.S. assignees of U.S. patents. ³⁴¹ They estimated an increase in patenting averaging between five and 30 patents per firm per year of participation, attributed to ATP.

In addition to estimating an increase in firm patenting due to ATP participation, Darby et al., concluded that ATP-funded organizations accounted for an unusually high percentage of all U.S. patents granted to U.S. entities during the period 1988–1996. They did not attribute this fact to ATP, but rather suggested it signals “the very top, leading edge technology firms are getting involved [with ATP] as well as the top universities.” ³⁴³

The Sakakibara-Branstetter study also examined the impact of participating in ATP projects on the overall research productivity of firms that participated in ATP-sponsored joint ventures, again using patent data as a proxy for productivity. ³⁴⁴ Their study compared ATP participants with a control group over a common time period, rather than taking the before-and-after approach of the Darby et al., study.

Sakakibara and Branstetter’s findings suggested that participation in ATP raises the research productivity of participating firms. ³⁴⁵ They found that participating in ATP joint ventures increased patenting in the targeted technology areas above the level of patenting that participating firms showed prior to participating in the project. They estimated that, at the margin, each instance of participation in an ATP project raised a firm’s research productivity, as measured by patents, by 8% per year. ³⁴⁶ They found evidence that consortia in which participating firms are technologically proximate were systematically more successful than consortia in which participating firms are involved in diverse technologies. ³⁴⁷

Reflecting the attention given in ATP’s authorizing legislation to small businesses, ATP’s evaluation program has addressed the following question: “Have small firms been able to compete successfully against larger firms for ATP awards?” Powell analyzed BRS data to help answer the question. The data showed that the majority of ATP participating companies, including subcontractors, are small, and that 61% of awards have gone to projects led by small firms. ³⁴⁸

Source: Powell, Business Planning and Progress of Small Firms Engaged in Technology Development through the Advanced Technology Program, 1999, p. 45.

“New technical knowledge must be used if economic benefits are going to accrue to the nation. This generally means the introduction into the market of a new product or process by the innovating firm, its collaborators, or other companies that acquire the knowledge.” ³⁵⁰

Innovating firms making progress toward commercialization may experience growth, increased capitalized value, higher sales, increased revenue, and return on investment. Their close collaborators and licensees are also positioned to make early commercial progress. The activities of the awardees, their collaborators, and licensees comprise ATP’s “direct path to impact.” ³⁵¹

Commercial Progress

Table 9–9 summarizes the commercialization progress of the first 50 completed ATP projects. Examples of the early technologies and the products and processes developed from them are given in Table 9–10. ³⁵² The individual project summaries in the Status Report provide accounts of each innovator’s specific progress toward taking their technology further into commercialization.

Table 9–9. Progress of Participating Companies in Commercializing New Technologies

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 17.

Employment Gains Among Small Innovating Firms

Company growth, as indicated by employment increases, is easier to isolate among ATP participants by examining small, single-applicant company recipients of ATP funding rather than large, multi-divisional firms, joint ventures, or nonprofit organizations. Employment changes in larger companies, non-profits, and joint ventures are too complex to link directly to the ATP project. But for small companies, “Rapid growth is generally a signal that the small innovating company is on the path to taking its technology into the market.” ³⁵³ The Status Report provides an overview of employment changes experienced by small companies included in the first 50 completed projects. Figure 9–3 shows the distribution of small companies by their percentage change in employment.

Figure 9–3. Employment Change at 31 Small Companies Receiving a Single-Company Award

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 18.

As the Status Report points out, “Sixty percent have at least doubled in size; four of them have grown more than 1000 percent.” ³⁵⁴

Returns to the Innovators

Returns to innovators can be measured in several ways. For the seven firms that carried out the tissue engineering projects—the technology focus of the RTI studies—the authors computed a composite measure of direct company returns. Although they calculated a measure of private returns for each project, to preserve proprietary information, they presented the results as a composite. The estimated composite internal rate of return (IRR) to the companies was 12%.

Consistent with ATP’s goal of generating broad-based benefits, the estimated composite private return was much lower than the composite social return of which it is a part. ³⁵⁵

Pelsoci’s case study of closed-cycle air refrigeration (CCAR) estimated revenues to private company innovators. ³⁵⁶ The study projected the present value of projected revenues from CCAR installations in the food service markets, volatile organic compound market, and liquid natural gas industries to total $65 million. (The IRR was not provided.)

Table 9–10. Examples of Products and Processes from the First 50 Completed ATP Projects

Source: Extracted from Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, Appendix A, pp. 253–258.

Finally, Ehlen’s case study of flow-control machining technology estimated increases in annual sales attributable to the project, focusing on three groups: (1) the main technology developer, Extrude Hone Corporation; (2) the first-line users of the technology, a group of aluminum casters; and (3) the automakers who would purchase the affected engines and install them in vehicles. ³⁵⁷

As Table 9–11 illustrates, for this process technology, the developer’s change in annual sales, shown in the bottom row, was relatively small. Again, the findings are indicative that the benefits to the innovator are much less than estimated social benefits.

Table 9–11. Impact on Industries of Near-Term, Five-Year Implementation Path for Flow-Control Machining Technology

Source: Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999, p. 45.

Collaboration Effects

One of ATP’s legislated mandates is to “aid industry-led United States joint research and development ventures...” Evaluation studies have variously focused on the structure and formation of new ventures, the role of universities in joint ventures, factors determining their success, their stability over time, and their benefits and costs. This section draws findings from 10 studies listed in Table 9–12. The discussion is organized along the lines of the sub-themes identified in the last five column headings of the table.

Table 9–12. Studies Extending Knowledge about Collaboration Activities

Collaborative Activity, New Venture Formation, and Attribution to ATP

Table 9–13. Summary of Study Findings on Frequency of Collaboration

The earliest survey of ATP participants by Solomon Associates, published in 1993, found that 46% of those companies interviewed after one year in the program had formed strategic alliances to further advance the technology associated with their ATP project. Some others were in the process of forming collaborative relationships, and others said it was “too early.” Only one company indicated no collaboration and no near-term plans to collaborate. ³⁵⁸

While the collaborative activities identified by Solomon were linked directly to ATP projects, the survey made no attempt at establishing a counterfactual comparison. Rather, it merely reported the frequency of collaboration associated with ATP projects, and captured some participants’ comments concerning the benefits.

Silber & Associates’ 1995 survey measured the frequency of collaborative activity, and also identified the nature of the collaborations, the number of partners, the degree of success, and the extent to which the relationships were new ones, “formed expressly to carry out the ATP project.” Silber concluded: ³⁵⁹

The ATP projects fostered new relationships among U.S. companies. Only one joint venture involved a consortium of companies that previously worked together. According to the participating companies, the others were formed expressly for the ATP. (p. 23)

The Silber survey specifically investigated the extent of collaborative activities among single-company applicant projects, finding that 52% of the single company applicants had formed collaborations and that they brought an average of four outside companies into their projects as subcontractors. The study found that each joint venture project included an average of six members, and 43% of the joint venture members “forged subcontracting relationships with an average of five additional companies.” ³⁶⁰

Powell and Lellock, from their analysis of BRS data, provided further evidence that collaborative activities are extensive among ATP projects, and that ATP played an important role in stimulating the collaborative activities. Analyzing data from 415 participants in 198 ATP projects—including single-company applicants and joint ventures—they reported that of the 86% of respondents reporting collaboration, 69% reported that ATP “to a great extent” was responsible for bringing about the collaboration. ³⁶¹ Powell and Lellock also found evidence of a substantial formation of strategic alliances aimed primarily at commercializing ATP-funded technologies. These included alliances with suppliers, customers, joint production, distributors, and licensing partners. ³⁶²

Feldman and Kelley’s comparisons of ATP award winners in 1998 with a control group of non-winners added to knowledge about the extent and importance of collaborative activity in ATP projects. The researchers asked winners and nonwinners alike about their partnering activities and whether their “most important collaborator” was someone with whom they had or had not partnered before. As indicated in Table 9–14, there was a high (79%) and equal tendency among winners and non-winners to propose projects entailing collaborative relationships. But the award winners were more likely to have a new collaborator as their most important research partner. Thus, of the 79% of winners whose proposals involved collaborative relationships, 59% were principally with organizations with whom they had not previously partnered, while for non-winners, 42% were with new partners. ³⁶³

Feldman and Kelley concluded that ATP successfully encouraged applicants to propose projects featuring collaboration, frequently with entirely new partners and, moreover, that ATP’s selection process advanced its goal of encouraging new venture formation by favoring the selection of proposals involving new partnerships. They provided the following pointed explanation of the importance of collaboration and the government’s role in making it happen: R&D collaboration is widely recognized as important to a firm’s R&D strategy for learning about technical advances in other organizations. But establishing new collaborative ties to other firms is difficult. An important role for government in public-private partnerships is to foster the formation of new R&D collaborations by reducing the costs to the firm of establishing these relationships. (p. 17)

Feldman and Kelley also tested whether the types of collaborative linkages differed among applicants to ATP who received and did not receive ATP funding. They found no statistically significant difference among winners and non-winners in their linkage to universities; that is, both groups were well linked to universities. But they did find that winners had more linkages to other businesses than non-winners. Based on their finding, Feldman and Kelley stated:

We interpret this difference to indicate that award-winning firms are better positioned than their non-winning counterparts to have their technologies taken up by other firms and to realize commercial success through a more developed network of ties to other firms. (p. 14.)

Table 9–14. Propensity to Collaborate with Other  Organizations and Form New Partnerships

Source: Feldman and Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, p. 20.

While the previous table shows collaboration at the proposal stage, Table 9–15 shows for the first 50 completed ATP projects the extent of collaborative activity over the entire project life, and the different types of collaboration. In a group comprised of only 16% joint ventures, 84% of the projects reported collaborative relationships—some for R&D, some for commercialization, and some for both. These statistics further indicate the extensive collaborative activities found in single-company projects. ³⁶⁴

Table 9–15. Collaborative Activity of the First 50 Completed Projects

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p 4. 367 Ibid., p. 5.

ATP’s authorizing language focused on the needs of America’s firms and industries. Following this intent, ATP focused its early program design and selection criteria on variables and relationships deemed salient to industry, treating universities as supporting players. But observing the behavior of firms that applied for ATP awards, it became apparent that collaborative relationships between firms and universities were more important than suggested by the original program language. Over time, it became evident from the proposals submitted to ATP that firms were choosing to include universities as R&D collaborators in major ways.

Several different studies investigated university representation in ATP projects, and they all found evidence of substantial university involvement. ATP’s Status Report provided an account of university involvement in the project case studies, and a count of the number of completed projects with university relationships. It reported that 48% of completed projects had university collaborations. ³⁶⁵

Feldman and Kelley’s exploration of 12 different types of linkages between ATP projects and universities, ³⁶⁶ for example, found that the entire large sample of 1998 applicants surveyed had ties to universities. Feldman and Kelly reported that 57% of all ATP projects had universities as joint venture members or subcontractors as of 1999. More often universities were in a subcontractor role. ³⁶⁷ Analysis of the first 50 completed ATP projects also showed the “close R&D ties of firms with universities.” Nearly half the projects (48%) involved universities. ³⁶⁸

Other university-related studies focused primarily on their roles in ATP projects and were covered in more detail in Part II, Chapter 4. These included the Hall et al., study ³⁶⁹ that found projects with university involvement are likely to experience more difficulty and delay, but also are more likely not to be aborted prematurely; the Kogut-Gittelman study ³⁷⁰ that found that firms with weak in-house research capabilities boosted their capabilities by allying with university scientists; and the Liebeskind study ³⁷¹ that found an inverted path of knowledge flow from industry to university.

In his case study of an ATP-funded joint venture, Link provided an up-close look at the formation and reformation of collaborators in the joint venture over a five-year project life. He showed that while the makeup of the collaborators changes over time, the roles remain relatively constant. ³⁷² Table 9–16 illustrates membership changes in the Printed Wiring Board (PWB) research joint venture.

The Silber survey provided statistical information on mid-course changes in project participants. The survey found that 59% of projects were carried out without changes in their collaborating organizations. It also found for “23 percent of the projects, at least one participating company was changed to a different company, and [for] 18 percent, at least one participant, along with that company’s piece of the project, was dropped altogether.” ³⁷³

Such changes in collaborative arrangements are important because they give rise to an issue for ATP management: at what point does a change in project makeup or associated goals no longer comply with the original criteria by which the project was selected for an ATP award? ³⁷⁴ Analyzing project changes helps management understand this issue better and is, therefore, a valid component of evaluation.

Dyer and Powell investigated factors increasing or decreasing the likelihood of success of collaborations. They interviewed company representatives and government project managers for 18 joint venture projects in the automobile industry, asking them a series of questions related to the success of their collaborative activities. They found that greater knowledge sharing and more effective coordination among participants characterized the more successful joint ventures. They developed hypotheses from the interviews about factors influencing the extent to which participants shared knowledge and influenced the costs of coordinating the venture’s activities, and expressed future plans to test their hypotheses through a survey.

The path diagram in Figure 9–4 shows the key factors participants said influenced the degree of success of their joint ventures, the direction and nature of impact, and the linkages from these factors to measures of outcome success. ³⁷⁵

Figure 9–4. Key Factors Influencing Collaborative Success

Starting on the left, the first set of circles contains five factors that participants identified as influencing their success in collaborating. Moving to the far right, the highlighted circle lists the main measures used by the participants in measuring their success. The center of the figure shows the key mechanisms through which the five factors on the left influence the success measures on the right. These factors are trust, information sharing, and coordination costs. The plus and minus signs along the directional lines show the nature of impact in the direction indicated. For example, geographic proximity of the members is shown to contribute to success in two ways: a positive effect on information sharing contributes positively to success; a negative (reducing) effect on coordination costs also contributes positively to success because the costs are negatively associated with success.

Source: Dyer and Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, 2001, p. 11.

Table 9–16. Membership Changes in the Printed Wiring Board Research Joint Venture, 1992–1996

Source: Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. 16.

According to participants in the Dyer-Powell study, ATP contributed to their joint venture success by (1) accelerating the development of high risk technologies, (2) increasing project stability, (3) getting projects through particularly difficult periods in their life cycles, (4) overcoming barriers to collaboration, and (5) increasing upfront planning.

Sakakibara and Branstetter also touched on the question of what determines consortium success in their research. ³⁷⁶ Looking at the impact of participation of companies in consortia on their patenting rates, the researchers stated:

… we demonstrated that there is a statistical link between a firm’s participation in an ATP project and that firm’s patenting in the technologies targeted by the ATP consortium … firms participating in consortia composed of other firms with similar patenting portfolios tend to do better. (p. 37)

The structure of collaborative arrangement is another factor important to the collaboration and the project’s overall success. Sketching a project’s collaborative structure can help project selection boards analyze key roles, responsibilities, and linkages among collaborators, and may help identify missing elements that threaten future project success. Ruegg depicted a variety of collaborative arrangements that have been used by ATP project participants, two of which appear in Figure 9–5. ³⁷⁷ The first shows a vertically structured joint venture and the second a hub and spoke structure with an important member missing, which likely adversely affected project success.

Assessing the Benefits of Collaboration

The 1992–2000 surveys of ATP participants investigated the perceived benefits of collaborating. The earliest survey conducted by Solomon Associates found that 80% of joint venture participants “discussed a myriad of benefits they had experienced to date because of the collaboration...” ³⁷⁸ But the study captured the nature of the benefits only in several anecdotal comments, such as:

In a consortium, everyone wins—lowest costs, highest quality processes for everyone in a high volume manufacturing environment. There is a higher probability of a successful outcome. (pp. 19–20)

Figure 9–5. Illustration of Two Project Structures with Alternative Collaborative Relationships

Source: Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, 1999, pp. 15–16.

Silber & Associates took the investigation of collaborative benefits further, compiling statistics on the extent to which companies benefited from the collaborative relationships of their ATP projects. This survey found that 60% of respondents benefited “to a great extent” from the collaboration, and 35% “to a moderate extent.” ³⁷⁹ The survey also tallied the frequency with which respondents said they experienced specified collaborative benefits—such as to stimulate creative thinking—to a “great, moderate, or small extent, or not at all.” ³⁸⁰

Powell and Lellock, in the BRS survey, continued to probe the benefits of collaboration. They largely repeated Silber’s probe of specific benefits from collaboration. Table 9–17 below compares results of the two studies’ findings on specific collaborative benefits. The first column lists the identified beneficial effects of collaboration in declining order of importance according to the BRS survey. The second column gives the frequency with which BRS respondents said each benefit was “significantly enabled by collaboration.” The third column gives the frequency with which the Silber survey respondents said they experienced the specified benefit “to a great extent from collaborating.” ³⁸¹

The same surveys that investigated benefits of collaboration also probed the costs. The earliest survey, by Solomon Associates, looked only at joint venture participants in the first competition. It found that after their first year, 60% of the joint venture participants identified one or more problems from the collaboration.

Almost all the problems were related to administrative issues, specifically auditing and accounting practices and legal arrangements regarding intellectual property, rather than with the research collaborations per se . ³⁸²

With a greater understanding of the extent to which single applicants were collaborating with others in ATP projects, Silber & Associates investigated collaboration problems across all project participants—not just joint ventures. The study identified a wider variety of problems than did the Solomon study, but the study did not identify the frequency with which the different kinds of problems were experienced. ³⁸³

Table 9–17. Specific Benefits of Collaborations

*The Silber survey’s closest matching category was called “increased customer acceptance.”

**The Silber survey’s closest matching category, for which the percentage applies, was called, “enabled you to develop technology while you engineered for volume manufacturing.”

The types of collaboration problems identified by the Silber & Associates survey were:

• Cultural differences between large and small companies; between companies and universities; and between people
• Different agendas and needs of collaborating organizations
• No single source of direction—“no general, only ten colonels”
• Lack of trust among collaborators; unwillingness to share information
• Length of time required to build successful relationships
• Challenge of identifying priorities, of deciding what and how to share technology
• Administrative effort required to manage the award

Some of the single applicants with collaborative arrangements implied that there are costs associated with formal joint venture collaborations by extolling the comparative advantages of their more informal arrangements. One cited the “single, clean decision-making chain...” the “better focus, better control, and increased flexibility with Single Applicant programs.” ³⁸⁴ In effect, they did not identify collaboration per se with increased costs, but rather with the jointventure form of collaboration. The freedom to apply to ATP as single applicants and use informal collaborative relationships to carry out proposed projects appears to be valued by firms.

Analysis of BRS data also provided evidence about the type and extent of costs associated with collaboration. Figure 9–6 shows the percentage of project participants who experienced (1) delays in beginning their R&D, (2) increasing project coordination and management costs, and (3) delays in product entry into the marketplace.  Note that only 9% reported that their costs were increased “significantly.” ³⁸⁶

Figure 9–6. Costs Attributed to Collaboration

Source: ATP Business Progress Reports for 974 applications being pursued by 463 companies in 256 ATP projects funded 1993–1997.

Both the Silber survey and the Powell and Lellock analyses of BRS data have suggested that costs associated with collaboration do enter into the decisions of ATP awardees but both have concluded that costs have not been widespread or serious overall in ATP. ³⁸⁷ The Silber study concluded, “For the majority of participants, however, the problems have been only minor stumbling blocks, resolved without serious consequences.” ³⁸⁸ Additionally, the study noted, “more than 90 percent of all participants said their collaborative relationships have worked out ‘extremely well’ or ‘fairly well.’” ³⁸⁹

Silber & Associates noted that 92% of respondents “said their ATP experience has piqued their interest in working collaboratively with other companies, and 96 percent of joint venture participants reported that this experience would influence them to engage in another joint venture down the road...” ³⁹⁰ This conclusion further indicates that the problems with collaboration in ATP projects have not been very serious.

Spillover Effects

The concept of economic spillovers has occupied a central place in the case for a public sector program like ATP, and has helped shape many of ATP’s program design features. Jaffe identified finding evidence of larger-than-average spillovers as a requirement for program success. Proof of large spillovers counters charges of corporate welfare, and supports public investment. Accordingly, a considerable portion of ATP’s evaluation program has been directed at explaining the concept of spillovers, and empirically estimating its presence and magnitude. This section draws material from the studies listed in Table 9–18. As may be seen from the table, nearly all the listed studies addressed either market or knowledge spillovers; only the first two addressed spillovers more broadly defined.

Table 9–18. Studies Informing Spillover Effects

*Denotes unpublished, draft reports.

Early Planning to Capture Spillovers

Mansfield’s pilot study for ATP, conducted in the early years of the program, explored the applicability of his model for estimating private and social rates of return from the commercialization of new products and processes by firms receiving ATP support. ³⁹¹

Mansfield’s study also highlighted the tasks that lay before ATP in attempting to adapt his model. As explained in greater detail in Part II, Chapter 4, Mansfield’s earlier applications of his approach used historical data, whereas ATP technologies were at best in the early stages of commercialization. His applications involved single products whereas ATP wanted to apply his model to technology platforms generating multiple products. Perhaps most importantly, his model sought to capture only market spillovers, whereas ATP was also interested in estimating knowledge and network spillovers.

Jaffe’s construction of a market and knowledge spillovers framework broadened the Mansfield social benefit framework. ³⁹² As earlier illustrated in Table 4–1, Jaffe identified three different sources of spillovers of relevance to ATP: knowledge spillovers, market spillovers, and network spillovers. In effect, ATP sought to increase and measure each of these forms of spillovers.

ATP’s evaluation program has focused on the first two, market and knowledge spillovers, and, at the time of this study, had not yet estimated network spillovers.  As defined earlier in Table 4–1, market spillovers occur when market dynamics cause some of the benefit of a product or process to flow to market participants other than the innovating firm, and knowledge spillovers occur when knowledge created by one agent is used by another without full compensation.

The data revealed by the BRS and the completed project status report strongly indicate that as a portfolio the funded projects are generating outputs that potentially will lead to both market and knowledge spillovers. ³⁹³ These data document publications, patents, patent citations, collaborative linkages, and products and processes—all from which spillovers can result.

Feldman and Kelley found that ATP is selecting projects with attributes conducive to generating large knowledge spillover effects. These attributes include linkages of proposing firms to other organizations, and positive attitudes of award winners toward information sharing and knowledge transfers to other firms. As Feldman and Kelley pointed out, “The more embedded a firm is in a network of such inter-organization ties, the more quickly the knowledge generated by the firm is expected to be absorbed by other organizations in the system.” ³⁹⁴ They found “...on average, award winners have a more extensive set of ties to other businesses than do non-winning applicants.” ³⁹⁵ They also found:

... a much higher proportion of award-winning firms exhibiting a tendency towards openness (30 percent), compared to non-winning applicants (19 percent). The higher rate of participation of such firms in ATPfunded projects suggests that the public interest is being served by enabling R&D activities that are more likely to generate knowledge which benefits both the participating firm and other firms not directly involved in the project. (p. 17)

Of the several evaluation studies that addressed aspects of spillovers, the Fogarty et al., study most directly focused on measuring knowledge spillovers. ³⁹⁶ This study was also notable for its methodological ingenuity. As explained in the treatment of the model in Chapter 8, the researchers applied systems analysis and fuzzy logic to analyze knowledge spillovers within networks of organizations.

To review, the method uses patent citations as a proxy for flows of scientific and technical knowledge. It uses citations to publications and earlier patents recorded on patent awards to construct networks of communication and influence among organizations within an innovation system. The direction and strength of relationships are determined by patterns of citing and sourcing among organizations.

One contribution of this study is that it constructs “networks” of organizations that extend beyond single interactions. The fuzzy logic and systems analysis components of the model are used to estimate the strength of the relationships among organizations. The model provides a far more detailed and comprehensive mapping of interactions of organizations important for the generation of knowledge spillovers than is possible in studying only interactions among members of a joint venture. For ATP projects generating pre-competitive, enabling technologies, the researchers identified the characteristics of R&D networks summarized in Table 9–19.

Table 9–19. Characteristics of R&D Networks

Although presented as an exploratory study with several of its key components yet to be fully tested, Fogarty et al., demonstrated use of the technique in identifying the knowledge spillover capability of an optical recording project funded by ATP. They showed that the project had an inherently high spillover potential and, hence, was a suitable selection choice for ATP.

Estimation of the magnitude and composition of knowledge spillovers also was a central objective of the Cohen and Walsh study. ³⁹⁷ This study advanced analysis of spillovers by its explicit attempt to link spillovers to appropriability; that is, the degree to which firms are able to protect the profitability of their own inventions and the strategies they use to appropriate the economic benefits from these inventions. The researchers noted that appropriability is affected by the mechanisms (e.g., patents, trade secrets) that firms use to appropriate economic rents ³⁹⁸ attributable to R&D, and that appropriability is a double-edged sword. On the one hand, a firm’s inability to fully appropriate the benefits of its own R&D reduces its incentives to invest in R&D; on the other hand, the inability of rival firms to fully appropriate the benefits of their own R&D constitutes a spillover to the first firm.

To review, Cohen and Walsh used data from the Carnegie Mellon survey to construct measures of appropriability, and then linked these data to other measures of intra-industry information flows, and firm and industry economic variables. Controlling for several variables, they found that intra-industry information flows complement firms’ own R&D efforts. The finding is consistent with the core propositions that led to ATP’s establishment and its key design features. In particular, by selecting generic technologies applicable to many firms both  upstream and downstream and by supporting appropriate joint ventures, ATP can foster the generation of knowledge spillovers, and thus increases the productivity of a firm’s R&D.

A variety of studies provide additional accumulating evidence of the potential of projects for large knowledge spillovers. The completed project status report, for instance, included patent citation trees showing how knowledge generated by ATP was taken up by other organizations. Figure 8–5 in Chapter 8 shows an illustrative patent tree for a patent from an ATP-funded project.

Documentation of extensive collaborative activities, reported earlier in this chapter, showed the establishment of linkages across organizations and persons through which information potentially can flow, generating knowledge spillovers. Fogarty et al., applied their new model to an ATP-funded optical recording joint venture, illustrating the approach and providing a more rigorous way of assessing these linkages.

Estimating Market Spillovers

Several ATP-funded evaluation studies have sought to estimate the magnitude of market spillovers of ATP projects. In one of these, a study by Research Triangle Institute (RTI), researchers imputed a measure of market spillovers for a portfolio of seven ATP-funded products in medical technology, as the gap between estimated social and private returns. ³⁹⁹ The modeling approach was presented in Chapter 6.

Table 9–20 shows the RTI study’s estimated composite social returns and private returns on investment for the seven projects examined in the study. Row (1) data are for the project as a whole. Row (2) data are based on just that part of the project’s social return attributable directly to ATP and ATP’s share of investment costs. Row (3) data apply to the project awardees. Row (4) data are the difference between (1) and (3). Row (5) data are the difference between (2) and (3). The market spillovers—the gap between social and private returns—are quite large. As might be expected, spillover estimates were particularly large for those cases where RTI researchers were able to estimate the value of changes in quality adjusted life years for patients from the new and improved medical treatments, in addition to treatment cost differences.

Table 9–20. Spillovers Imputed by Comparing Composite Social Returns, Public Returns, and Composite Private Returns on Seven Tissue Engineering Projects

Source: From Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, Tables 1–3 and 1–5

Based on these results, the authors concluded:

The wide disparity between social and private returns indicates the importance of ATP incentives to the private sector to pursue these technologies. Because the social returns far outweigh the returns to the companies developing, commercializing, and producing these high-risk projects, the private sector may under invest in these kinds of high-risk projects.

Another study that estimated market spillovers is Pelsoci’s study of CCAR technology, estimated as the difference between social and private returns. ⁴⁰⁰ After working through the technical performance features of the new technology and identifying pathways to market, the study estimated social and private returns based upon a conservative base case and an alternative optimal scenario. Like the RTI study, the Pelsoci study found a large gap between the estimated social and private, suggesting a large spillover effect. The estimates are for market spillovers only, because the estimated benefits are derived from sales of CCAR units.

Austin and Macauley used their cost index approach to estimate market spillovers from new technologies. ⁴⁰¹ Applied to the development of two digital data storage technologies funded in part by ATP, the model yielded an estimate for consumer welfare gains over 5 years of $1.5 billion and $2.2 billion, respectively, for each of the two technologies.

State and International Technology Programs

Prior to the establishment of ATP in the United States, several industrialized countries already had in place similar technology development programs. Within the United States, the legislative creation of ATP in 1988 followed the establishment of nominally related programs by several states, and came near the peak of an approximately 10-year period when technology development programs were adopted by a number of states. The experiences of other nations’ public-private partnership programs and those of the American states were of keen interest to ATP. Program design features, operational issues, and program evaluation are among topics of common interest. In addition, in the case of state governments, ATP has sought to identify ways in which its specific features could be effectively integrated with those of state governments.

From the five studies listed in Table 9–21, this chapter draws findings on how ATP compares with programs of other countries, and how it relates to state run programs.

Table 9–21. Studies Comparing State and International Programs

Comparing ATP with state programs and programs abroad fulfils several specific objectives, including: (1) coordination of federal and state activities, (2) distillation of best practices from comparable programs, (3) compliance with legislative requirements on international reciprocity and national treatment, and (4) provision of a test bed for new evaluation models by drawing data appropriately from programs with a longer history.

Programs in Other Countries

There were two objectives in ATP’s examination of counterpart programs in other countries: (1) to learn from them and make ATP a better program, and (2) to implement a statutory requirement that ATP consider proposals from U.S.-based subsidiaries of foreign-owned companies only if they passed three tests beyond those applied to proposals for U.S.-based firms. One of these tests essentially requires national treatment, namely that the parent company is incorporated in a country that affords U.S.-owned companies the opportunity to participate in government-funded joint ventures comparable to those of any other company in that country. Enforcing this provision required ATP to keep abreast of counterpart programs in other nations.

Chang’s framework and lexicon for identifying the key program features of technology development programs provided the structure to accomplish both of the above objectives. ⁴⁰² She identified analogues of ATP “in many countries of varying sizes and economic maturity.” ⁴⁰³ Illustrated in Table 9–22 is her comparison of ATP with similar programs in Canada, the European Union, Finland, Japan, and the United Kingdom. The basis of comparison is their year of launch, mission, technical scope, eligibility regarding project lead, stage of research, level of formality of selection process, and cost-share requirement.

Table 9–22. Comparative Features of ATP and Its Analogues

Source: Chang, A New Lexicon and Framework for Analyzing the Internal Structures of the U.S. Advanced Technology Program and its Analogues Around the World, 1998.

The table compares ATP with similar programs in Canada, Finland, Japan, United Kingdom, and the European Union in terms of the seven features listed.

Programs in Japan and Israel have figured more strongly in ATP’s evaluation than others. This is because several studies—those by Sakakibara and Branstetter and Griliches et al.—looked to these two countries for datasets that covered more years of operation than ATP. They used the data to provide a test bed for assessing the feasibility of specific evaluation methods for which adequate data from ATP was lacking at the time. In addition, these comparative studies also served to condition expectations about the timing and direction of project impacts.

Sakakibara and Branstetter used patent data from Japan to examine the long-term economic impacts of consortia-based joint ventures. ⁴⁰⁴ Having long-term data is essential to determining the impacts of consortia arrangements, because, as their findings indicate:

… much of the impact of research consortia is felt long after the inception of the project. In fact, evidence from the Japanese consortia suggests that some of the strongest effects are felt after the official cessation of the consortia. What this means is that the relatively short time series of data available on U.S. consortia will tend to underestimate the total impact of the consortia. (p. 6)

In a similar vein, Griliches et al., drew upon Israel’s over twenty years of experience with ATP-type programs to address central issues surrounding ATP’s establishment; that is, whether government support of private sector R&D served as a substitute for or complement to private sector R&D, and how it affected productivity. ⁴⁰⁵ The study served both to highlight the strengths and limitations of econometric modeling, for it performed quantitative tests of a salient policy issue while indicating the sensitivity of findings to model specification and sample selection. Their specific finding was that government-supported R&D generated high rates of firm productivity increases, but also noted that this conclusion “should be treated cautiously” because of potential problems in model specification, exclusion of relevant variables, and selection bias.

The essentially contemporaneous development of ATP and state technology development programs created both opportunities and complexities in coordination. A key question in the early years of the program was the character of the relationship between ATP and the state programs. Unlike the Manufacturing Extension Partnership Program, also established in the Omnibus Trade and Competitiveness Act of 1988 and located at NIST, there were few formal linkages between ATP and the state programs.

A two-volume ATP report, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, examined the character   and potential of ATP and state program linkages. Volume 1 of the study, authored by Schachtel and Feldman, analyzed the structure of state programs, describing the range of activities supported by these programs, and indirectly pointing to the  types of market failures that these programs were intended to offset. ⁴⁰⁶ Drawing on a model of product commercialization development by Goldsmith, the study illustrated various combinations of private-sector and public-sector relationships.

Analysis of the inventory of private-sector and public-sector relationships led to the following conclusions: State technology programs spanned the research and development continuum, ranging from an emphasis on basic research and human capital development, to pre-competitive research, to spin-off firms and product development. Some of the larger state programs provided support for each of these activities. In the main, though, state programs clustered around the downstream applied/commercialization segment, rather than the upstream research segment of the continuum. ATP, in contrast, centered its activities on technical challenges, supporting work primarily in the concept and development phases. The comparison of ATP’s focus with that of state programs is illustrated in Figure 9–7, using a version of Goldsmith’s nine-cell model that Schachtel and Feldman used in Table 4–5 to categorize state programs.

Figure 9–7. Comparison of the Focus of ATP and State Technology Development Programs

Source: Adapted by Ruegg and Feller from Goldsmith.

The Schachtel-Feldman study also highlighted the possibilities of firms combining support from both ATP and state governments, drawing upon ATP support, as above, for concept development, and then coupling this support with state support for downstream scaling up and product development support.

Volume 2 of the study, authored by Feldman, Kelley, Schaff, and Farkas, provided four case histories of the evolution of “technology-pioneering” firms that received support from ATP as well as state programs. ⁴⁰⁷ Of interest is the firm’s perspective of identifying and applying for various types of support from multiple sponsors. Although too limited in number to permit generalizations, the four cases do highlight several ways ATP and state programs augment each other. Consistent with the analytical framework noted above, ATP (and other federal agencies) were found to play the largest direct role in funding the upstream, concept development R&D, while the state programs were found to fund more of the downstream commercialization work. In the words of the researchers:

In sum, public resources have played a critical role in the ability of technology pioneering companies to develop their technologies and their businesses.

Moreover, the cases illustrate how state and federal government resources complement one another. (p. vi)

Measures of Overall ATP Performance

Empirically derived, comprehensive measures of a program’s ultimate outcome are a holy grail of evaluation. But, in the case of a young program like ATP where the long-term outcome is still unfolding, indicators of progress, partial measures of benefit, and assessments of effectiveness to date are reasonable, and even ambitious, evaluative accomplishments.

While some of the evaluation studies of ATP over the past decade focused on single issues, others attempted to provide broader assessments of the program. This section draws findings from the 13 studies listed in Table 9–23 that bear on  the overall performance of ATP—its impact on national industrial competitiveness and the national capacity to innovate, its ability to deal appropriately with failed projects, and measures of portfolio performance, social returns and overall program effectiveness.

Table 9–23. Studies Providing Broader Assessments of Program Performance

ATP’s Contribution to Project Impacts

It is not enough to know that the projects cost-shared by ATP have the desired impacts. To evaluate ATP’s contribution, evaluators must determine to what extent ATP caused these impacts. Did federal funds leverage or substitute for private funds? Might participating organizations have abandoned, delayed, or reduced the scale or scope of their efforts, or would they have taken the same steps without ATP? Since it is not unusual for a program to take full credit for any benefits that appear to emanate from it, the attribution requirement for ATP sets a comparatively high assessment standard in the sphere of applied evaluation.

Most ATP evaluation studies examined in this report have attempted to move beyond documentation of positive outcomes of one form or another associated with ATP funding to more formal tests of attribution and causation. The main techniques have been counterfactual analyses and control groups. These techniques and their application in ATP evaluation studies were discussed in Chapter 4. An earlier section in this chapter summarized evidence from studies showing that ATP has encouraged and made it possible for businesses to take on projects with higher technical challenges, start them sooner, carry them out faster, and enlarge their scope. In addition, another summary was provided of evidence that ATP has stimulated new collaborative activity. The evidence cited in these two earlier sections provides a backdrop for examining the broader measures of performance that have been developed for ATP and its portfolio of projects.

Improving the Industrial Competitiveness of the United States

By partnering with U.S. businesses and encouraging their long-term followthrough with new technologies, ATP seeks directly to affect the likelihood that U.S. companies will pursue the funded innovations. By funding broadly enabling technologies and promoting the dissemination of knowledge created, ATP further seeks to affect the competitive capabilities of U.S. companies.

While there are no aggregate, program-wide measures of ATP’s success in improving the competitiveness of the nation, there is a growing mosaic of evidence. The following study findings relate to ATP’s effect on competitiveness.

The first insight to the program’s effects on competitiveness came from the first survey of project participants, conducted by Solomon Associates. ⁴⁰⁸

When asked whether participation in the ATP had ‘changed your organization’s competitive standing in any way’, vis a vis domestic and foreign competitors, the majority (58 percent) maintained that it was ‘far too premature’...Nonetheless, nearly a quarter of the participants (23 percent) did state that participation in the ATP program had changed their competitive standing in the marketplace [with the change implied as positive]. (p. 21)

Three years later, the percentage of participants stating that their competitive standing had improved since they won the ATP award was 72% versus the 23% found earlier. Of the 72%, nearly half attributed the improvement to their ATP award “to a great extent” and another 28% “to a moderate extent.” Some of the nearly 30% who reportedly had not yet experienced any change in their competitive standing still hoped to: “We have high hopes the technology will increase competitive standing—potentially the ATP award will have a significant impact.” ⁴⁰⁹

The most recent findings on competitive position comes from the analysis by Powell and Lellock of BRS data, ⁴¹⁰ but the data are not strictly comparable with the earlier survey data. The BRS approached the question by collecting data by application area rather than by project, such that a project with two application areas might experience gains in competitive standings in the one and losses in the other. An analysis of the BRS data, approximately seven years after Silber’s survey, showed 32% of participants reporting that their overall competitive standing had improved, and 9% that it had worsened, with respect to all applications of their ATP-funded projects currently on the market or being planned.

The earlier surveys may have captured primarily the views of companies in the pre-commercialization phase when they were receiving greater recognition and higher standing in their industry due to their ATP awards, whereas the more recent BRS survey may better capture the realities of the marketplace. As stated by Powell and Lellock ⁴¹¹ in the later data analysis:

These results suggest that some ATP-funded companies face highly competitive, dynamic product markets in commercializing their technologies. The entry of ATP-funded technologies into the marketplace will tend to increase the level of competition in existing markets as increased investment and advances by ATP-funded firms stimulate others to “catchup” or to make their own advances. ATP funding also helps increase the visibility of promising new technology areas, and this helps spawn entirely new firms and product markets. (pp. 40–42)

Link presented evidence that ATP affected change in the entire printed wiring board industry. ⁴¹² This finding is likely more significant in terms of evidence that ATP is meeting its legislated goal than the previously presented finding that ATP affected the competitive position of a certain percentage of project participants. The special significance of industry-wide competitive gains is that they more clearly represent a gain for U.S. industry in international competition.

Members of the joint venture reported that as a result of the project, “the domestic PWB industry as a whole has increased its competitive position in selected world markets...” ⁴¹⁴ They also reported:

... the PWB project has increased industry’s share in every market segment, with the strongest positive responses in the computer and military segments. No member [of the joint venture] was of the opinion that they or other members of the joint venture had increased their share at the expense of nonmembers, and this can be attributed to the fact that the results of the PWB project have been widely disseminated. (p. 31.)

According to Link, at a meeting near the end of the project, the president of the National Center for Manufacturing Sciences “credited the project with saving the PWB industry in the U.S. with its approximately 200,000 jobs.” ⁴¹⁵

Increasing the National Capacity to Innovate

A number of the evaluation findings suggest that ATP is adding to the nation’s capacity to mount future innovative activity. That is, the funded projects appear to build infrastructure important for innovation, and it remains after the funded projects are complete. This infrastructure consists of strengthened companies with more advanced capabilities and with more scientific and technical personnel with new knowledge and experience. It exists in new linkages among companies, universities, government laboratories, state agencies, and other organizations. It is evident in the newly created knowledge residing in people, journals, books, lab notes, and patents; and embodied in new products and processes. This infrastructure is critical to the nation’s capacity for further innovation.

The study by Darby et al., which investigated “whether ATP projects have a general effect on formation of new intellectual property within the firm,” ⁴¹⁶  suggested that ATP is adding to the nation’s infrastructure important to future innovation:

In our view, ATP not only provides funding awards to participants, but also promotes “institution-building” in the process, encouraging applicants to establish new organizational structures that facilitate innovation and the capture of inventions in technologically advanced commercial products. Institution building takes place in ATP in a number of ways. First, ATP supports firms willing to experiment and develop approaches that are novel and at the technological frontier. ATP stimulates industry to initiate projects that are higher in risk, with greater potential for broader economic impact. Second, ATP encourages cooperation and collaboration in R&D activities, among JV partners, and also through subcontracting relationships with universities, firms, and other organizations. ...In social science terminology, the ATP project changes participants “social embeddedness” in networks of relations with other firms and organizations. (pp. 4–5)

A nation’s capacity to organize resources efficiently and to mount major new research activities effectively is a national resource of incalculable value. If, as the studies suggest, ATP is contributing to the nation’s capacity to innovate, this effect could be the single most important impact of ATP.

Dealing with Failed Projects

It is clear from the body of evaluation studies examined that not all ATP projects are successful—nor might they be expected to be if ATP is meeting its mandate of funding high-risk research. While failure in some projects should be expected, the rate of failure and how the program deals with project failure are questions that bear on program management and performance. A comprehensive failure analysis has not yet been undertaken for ATP at the time of this study, but several studies shed light on the issue of project failure and how ATP handled it.

In theory, projects fall on a continuum ranging from full success to complete failure, but in practice, few projects fit either extreme. Most achieve varying degrees of success, with something generally being learned even from projects that fail to start. As noted in the completed projects status report, which contains a section on terminated projects: ⁴¹⁷

... terminated projects may yield patents, papers, collaborative relationships, and products. ... [They] entail a great deal of integrated planning for research, development, and business activities. ... [They] entail substantive cross-disciplinary contact among scientists and other researchers, cross talk among technical and business staff, and high-level negotiations among business executives at different companies. ... Often the planning period brings together business staff with university researchers, federal laboratory specialists, and other nonprofit facilities. There are likely to be extended effects of this process that may bear fruit in future diverse and difficult-to-capture ways. (p. 260)

It is apparent from the body of evaluation studies that ATP does not simply declare all projects successful because they deliver some minimal level of knowledge. That said, projects that never start, that are terminated before completion, or that show no or few outputs are generally deemed failures.

According to the completed projects status report, approximately 5–6% of all ATP projects funded over the program’s first decade were terminated after the award announcement and before completion. ⁴¹⁸ Figure 9–8 shows the reasons for termination. ⁴¹⁹ Of the 16 first terminated projects, five were joint ventures whose members failed to reach agreement among themselves. Because the participants were unable to form their intended collaboration, ATP was unable to form the intended partnership with the joint venture, and, therefore, the projects never got off the ground. Eleven other projects were stopped prior to their completion. No ATP funding was awarded to those projects that never started, and only partial  funding went to those that started but were terminated, the percentage depending  on how long the projects ran.

Figure 9–8. Distribution of Terminated Projects by Reason for Termination

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, p. 260.

Other identifiable poor performers are those receiving the lowest ratings of the Composite Performance Rating System (CPRS), discussed in Chapter 8. Based on CPRS ratings, nearly a quarter of the first 50 completed ATP projects were poor performers. ⁴²⁰

To be sure, the CPRS is a rough and approximate rating metric, but its use and the results show that ATP discriminates among its projects in terms of rating their level of success, and has a management tool for identifying the relative progress of completed projects in progressing toward program goals.

Measures of Progress, Social Benefits, and Overall Program Effectiveness

The body of evaluation studies completed over the past decade includes specific measures of the social impact of individual projects and, taken as a group, supports the formulation of general conclusions about the success of ATP. This section looks at studies that cut across multiple themes and program goals to suggest how ATP has performed at large.

Distribution of Projects by Overall Progress Ratings

The extensive data on outputs and outcomes have been analyzed and presented piecemeal to indicate project progress toward achieving particular goals. But, as noted, the data have also been combined and used to rate project performance in the overall achievement of ATP’s goals. Figure 9–9 shows the distribution of the first 50 completed projects by their overall CPRS performance scores. Figure 9–10 further shows the distribution of projects by their overall performance scores broken into five technology areas.

Figure 9–9. Distribution of Projects by Overall Performance Score

Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, p. 20.

Although they comprise only about 10% of total projects funded by ATP during its first decade, the first 50 completed projects provide a snapshot of portfolio performance free of project selection bias. As might be expected, the top performers, signaled by 4 stars, were limited to a relatively small group—16% of the total. Another 26% were in the next 3-star tier, also representing robust performance. Again, as might be expected, the largest group, 34%, were in the 2-star tier, exhibiting a moderate level of progress overall. As was noted in the previous section, the bottom performers comprised 24% of the total. ⁴²¹

Figure 9–10. Distribution of Projects by Overall Performance Score and Technology Area

Note: When divided into technology areas, the numbers of projects in the sample in each area are too small to support drawing general conclusions about the performance of ATP projects by technology area. Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 21.

Measures of Social Benefits

As ATP’s first decade drew to a close, evidence began to mount that “Estimated benefits attributed to ATP from just a few of the top-performing 50 projects ... far exceed the total ATP costs for all of the 522 projects funded to date [through 2000].” ⁴²² This conclusion was based on estimated social benefits drawn from a collection of detailed case studies, together with implied benefits based on the less detailed “status reports” performed for the completed projects. Table 9–24 shows a compilation of measures of social benefits drawn from case study. The measures are partial, in that they do not cover all expected applications of the technology and do not include an estimate of the economic value of knowledge spillovers.

Additionally, the approach by which the estimates are derived is similar, in that they are based on benefit-cost methodology, but different in terms of their varying assumptions, time periods covered, base year in which dollars are stated, and specific modeling techniques.

Table 9–24. Measures of Economic Benefit from Six Detailed Case Studies

The largest share of estimated social benefits supporting the above conclusion comes from RTI’s projections for three top performing projects that developed medical technologies. Benefit estimates included $98 million, $47 million, and  $15 billion, from biocompatible polymers for cartilage repair, stem cell replication, and biomaterials for prostheses, respectively, all attributed to ATP. ⁴²³ The set of case studies provided evidence that ATP is funding a portfolio of projects whose benefits extend well beyond the direct award recipients. A major limitation was the prospective nature of the case studies.

Some of the detailed case studies did not so clearly separate out what are estimated private returns and what are social returns. Link, for example, used estimated R&D costs savings as a minimum estimate of project benefits. He estimated that the printed wiring board joint venture saved a minimum of $35.5 million in R&D costs, but did not attempt to apportion these estimated cost savings (proxy for benefits) between the awardees and other potential beneficiaries. ⁴²⁴

Beyond the estimated impact of the first 50 completed ATP projects, Ehlen’s study estimated an annual increase in GDP from the ATP-funded flow-control  machining technology ranging from $142 million to $1.9 billion, depending on the assumed implementation path. He estimated that annual federal tax revenue would increase by between $34 million and $527 million. Pelsoci expected net benefits to result from “improved food safety, higher food processing yields and production rates, improved quality of processed foods, reduced harmful environmental emissions, additional U.S. exports, and crossindustry knowledge diffusion about ATP-funded innovations.” ⁴²⁵ He estimated net present value economic benefits from the ATP-funded closed-cycle air refrigeration technology between $459 million for a conservative case and $585 million for a more optimistic case.

The status report case studies for the stronger group of completed projects also provided evidence of growing revenue and potential benefits. For example, use of scalable parallel programming technology developed by Torrent Systems was reported to have allowed an early user to increase its revenue by $100 million annually. ⁴²⁶ Although these shorter cases do not provide quantitative estimates of net social benefits, except as they are drawn from previous in-depth economic case studies, they point toward future net benefits.

The NRC has provided the most comprehensive assessment of ATP to date. Drawing on expert judgment, informed by a large body of studies about ATP, and further informed by its broader investigation of federal public-private partnership programs, the NRC committee responsible for the ATP assessment concluded: ⁴²⁷

... [T]he Advanced Technology Program is an effective federal partnership program ... achieving the goals ascribed to the program in the Omnibus Trade and Competitiveness Act of 1988. (pp. 87–90)

Addressing the issue of ATP’s success in advancing important societal goals, the NRC committee concluded:

The selection criteria applied by the program enable it to meet broad national needs and help ensure that the benefits of successful awards extend across firms and industries. Its cost-shared, industry-driven approach to funding promising new technological opportunities has shown considerable success in advancing technologies that can contribute to important societal goals such as improved health diagnostics (e.g., breast cancer detection), developing tools to exploit the human genome (e.g., protection against colon cancer), and improving the efficiency and competitiveness of U.S. manufacturing. (p. 5)

Addressing the issue of whether ATP’s project selection process helps it to avoid displacing private funding sources, the NRC committee concluded:

The program’s peer review of applicants for both technical feasibility and commercial potential supports its goals of helping advance promising new technologies that are unlikely to be funded through the normal operation of the capital markets. (p. 6)

The NRC committee also examined ATP’s evaluation program, concluding:

The program has set a high standard for assessment involving both internal and independent external review. The quality of this assessment effort lends credence to the program’s evaluation of its accomplishments. (p. 6)

With respect to its review of the extant body of evaluation studies, the NRC committee concluded:

The extensive assessments of the program show that it appears to have been successful in achieving its core objectives, that is, enabling or facilitating private sector R&D projects of a type, or in an area, where social returns are likely to exceed private returns to private investors. (p. 6)

In addition to its findings, the NRC study made recommendations for “a series of operational improvements designed to make the program more effective.” Among the committee’s recommendations were two that relate to evaluation; namely, that outside assessments be released early to the research community, and existing efforts be enhanced to integrate assessment results into the ATP decision process. ⁴²⁸

Summary of Crosscutting Findings

Study evidence also points to the achievement of the specific sub objectives prescribed in its enabling legislation. ATP’s fostering of collaboration has been documented by survey, case study, and econometric studies. ATP’s attention to small businesses, and the success of small businesses in the program are supported by substantial study evidence. Study documentation of publications, papers, patents, citations by others, extensive collaborative relationships, and awards by third parties for technical contributions of projects all point to ATP’s success in stimulating the generation and dissemination of new scientific and technical knowledge.

Findings about spillovers are in line with theoretical requirements for the program—that they are ample and much in excess of private returns. Given that the study focus over the first decade was on measuring the value of market spillovers, the total stands to be much greater when estimates of knowledge spillovers are added to the market spillovers.

Finally, the extensive use of counterfactual and control groups in the evaluation studies of ATP has provided a growing body of rigorous evidence that ATP is a potent causal factor in the observed impacts. Repeatedly, study findings have shown a leveraging rather than substitutive effect of ATP funding on private sector support of R&D.

295 Competitiveness effects are reported in the last section of this chapter rather than here, because the thrust of the legislated goal of the program is towards the overall competitiveness of U.S. firms relative to foreign firms and not the competitiveness of individual firms.

296 Branscomb et al., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000, p. 2.

297 Ibid., p.6.

298 David Morgenthaler, “Assessing Technical Risk,” in L. M. Branscomb, Kenneth P. Morse, and Michael J. Roberts, eds., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000, p. 106.

299 L. M. Branscomb and Philip E. Auerswald, Between Invention and Innovation: An Analysis of the Funding for Early-Stage Technology Development, NIST GCR 02–841 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

300 Gompers and Lerner, Capital Formation and Investment in Venture Markets: Implications for the Advanced Technology Program, 1999, p. 20.

301 Ibid., p. 39.

302 Ibid., p. iv.

303 Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 11.

304 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001.

305 Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999, p. 4.

306 David Ayares, “The Five Little Pigs: A Tale of Xeno-Organ Transplant. In Charles W. Wessner, ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000).

307 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, p. 29.

308 Jeanne W. Powell and Karen Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 30.

309 Ibid., p. vi.

310 Ibid., p. 31.

311 Ibid., p. 11.

312 Feldman and Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, pp. 18–19.

313 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 6.

314 Ibid., pp. 10–11.

315 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 41–43.

316 Ibid., p. 42.

317 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 31.

318 Ibid., p. 31.

319 Laidlaw, Acceleration of Technology Development by the Advanced Technology Program: The Experience of 28 Projects Funded in 1991, 1997, p. 28.

320 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, pp. 36–41.

321 Maryann Feldman and Maryellen Kelley, “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in Charles Wessner ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000).

322 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 7.

323 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 37–40.

324 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, pp. 11–12.

325 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 29.

326 Laidlaw, Acceleration of Technology Development by the Advanced Technology Program: The Experience of 28 Projects Funded in 1991, 1997.

327 Ibid., pp. 15–16.

328 Ibid., pp. 22–23.

329 Ibid., p. 25.

330 Ibid., pp. 28–29.

331 Ibid., pp. 28–29.

332 Ibid., p. 25, pp. 34–35.

333 Ibid., pp. 25–37.

334 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, pp. 1–23.

335 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. iv.

336 The analyst was unable to obtain responses for six of the 50 projects.

337 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 24.

338 Already discussed in Chapter 4 and Chapter 7 was the work of Griliches, Regev, and Trajtenberg with Israeli program data, which showed a positive effect of technology support programs on firm productivity.

339 Darby et al., Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes, 2002, p. 10.

340 Ibid., p. 13.

341 Ibid., p. 19.

342 Ibid., p. 1.

343 Ibid., p. 19.

344 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002.

345 Ibid., p. vi.

346 Ibid., p.vi

347 Ibid., p. vi.

348 Powell, Business Planning and Progress of Small Firms Engaged in Technology Development through the Advanced Technology Program, 1999, with the percentage participation figure updated by Powell in 2002.

349 Ibid., p. 45.

350 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p.17.

351 Knowledge spillovers to others and their subsequent commercial activities provide an indirect path to project impact. For examples of ATP’s direct and indirect paths to impact, see Ruegg, “Delivering Public Benefits with Private-Sector Efficiency through the Advanced Technology Program,” 2000, pp. 118–120.

352 Ibid., Appendix A, pp. 253–258.

353 Ibid., p. 18.

354 Ibid., p. 18.

355 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, pp. 1–23–1–24.

356 Pelsoci, Closed-Cycle Air Refrigeration Technology for Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. ix.

357 Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999.

358 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 19. comparison. Rather, it merely reported the frequency of collaboration associated with ATP projects, and captured some participants’ comments concerning the benefits.

359 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996.

360 Ibid., p. 25.

361 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 19.

362 Ibid., p. 28.

363 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 19–20.

364 R. Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” in Charles F. Wessner, ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000) pp. 261–262.

365 Ibid, pp. 261–262.

366 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 12–13.

367 Ibid., p. 5.

368 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 4.

369 Hall et al., Universities as Research Partners, 2002.

370 Kogut and Gittelman, Public-Private Partnering and Innovation Performance Among U.S. Biotechnology Firms, 2000.

371 Liebeskind, Study of the Management of Intellectual Property in ATP-Grantee Firms, 2000.

372 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997. ATP allows time for projects to find replacement partners when they lose critical project participants.

373 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 33.

374 Responding to project changes requires balancing the need for flexibility to allow firms to make changes needed for project viability, with the need to adhere to ATP’s legislated mandate to fund high-risk research to develop technologies with potential for generating broad-based benefits. To protect the public trust, ATP decides on a case-by-case basis, after reviewing changes in project makeup, whether to approve or disapprove the changes.

375 Because of the relatively small sample size, their findings should be regarded as suggestive and tentative.

376 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002.

377 Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, 1999, pp. 9–17.

378 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, pp. 19–20.

379 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 24.

380 Ibid., pp. 27–30.

381 There is not an exact correspondence between all categories of benefits probed by the two surveys. The BRS used the ratings: “significantly, moderately, little/none, not sure.” Silber used the ratings: “great extent, moderate extent, small extent, no extent.” The comparison shown in Table 9–16 assumes that “significantly” and “great extent” are roughly equivalent.

382 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 20.

383 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 31–32; the list of problems is summarized from Silber’s discussion.

384 Ibid., p. 34

385 Ibid., p. 36.

386 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 22.

387 In response to early evidence of joint venture.

388 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 31.

389 Ibid., p. 33.

390 Ibid., p. 33.

391 Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996.

392 Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, 1996.

393 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000; and Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001.

394 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, p. 11.

395 Ibid., p. 14.

396 Fogarty et al., ATP and the U.S. Innovation System—A Methodology for Identifying Enabling R&D Spillover Networks with Applications to Micro-electromechanical Systems (MEMS) and Optical Recording, 2000.

397 Cohen and Walsh, R&D Spillovers, Appropriability and R&D Intensity: A Survey Based Approach, 2000.

398 Economic rent refers to the excess of return over a competitive rate of return.

399 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998.

400 Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002.

401 Austin and Macauley, Estimating Future Consumer Benefits from ATP-Funded Innovation: The Case of Digital Data Storage, 2000.

402 Chang, A New Lexicon and Framework for Analyzing the Internal Structures of the U.S. Advanced Technology Program and its Analogues Around the World, 1998.

403 Ibid., p. 67.

404 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity ofParticipating Firms: A Framework Using Both U.S. and Japanese Data, 2002.

405 Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, 2000.

406 Schachtel and Feldman, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. 1, 2000, p. 1.

407 Feldman et al., Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. 2, 2000.

408 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 21.

409 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 46–47.

410 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000.

411 Ibid.

412 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997.

413 Ibid., p. 6.

414 Ibid., pp. 30–31.

415 Ibid., p. 30.

416 Darby et al., Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes, 2002.

417 That a terminated project may yield knowledge gains even if it fails to achieve its ultimate goals is demonstrated by a case study of a terminated project provided in Advanced  Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, pp. 261–265.

418 Ibid., p. 260.

419 Ibid., pp. 259–260.

420 Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” 2000, pp. 259–277.

421 Advanced Technology Program, Performance of 50 Completed ATP Projects Status Report 2, 2001, p. 29–21.

422 Ibid., p. 25.

423 Ibid., p. 21–22.

425 Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, pp. vii-ix.

426 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, pp. 22–23.

427 Wessner, The Advanced Technology Program: Assessing Outcomes, 2000.

428 Ibid., p. 94.

429 Chapter 9 in its entirety is a summary of the findings of many studies. Hence, this end-of-chapter summary provides a higher level overview of the results of the chapter.

Return to Table of Contents or go to Chapter 10: Conclusions and Recommendations.

Date created: July 15, 2004
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)