for Evaluating Public R&D Investment
CHAPTER 9: A Crosscutting Look at Study Findings
This chapter rotates the analytical perspective found in Part II to focus on the findings of the evaluation studies supported by ATP. It highlights the general principle articulated in Chapter 2: that taking multiple analytical and empirical perspectives and evaluation approaches to complex questions contribute to more reliable, credible, and acceptable findings.
This crosscutting analysis of ATP’s evaluation studies is organized around the following major themes: (1) firm/industry effects, (2) collaboration effects, (3) spillover effects, (4) comparisons and interfaces with other programs, and (5) measures of overall ATP performance, including portfolio analysis, social returns, and impacts on competitiveness. A table at the beginning of each thematic section presents a listing of reports from which the findings are drawn and provides a further breakdown by sub-themes treated in the section.
Direct Private Firm Effects
Private firms play a central role in ATP’s operations. As Spender put it, “ATP harnesses private firms’ resources to the public interest without seeking to outguess market forces.” 294 The term “public-private partnership” further recognizes that the program works only in conjunction with private-sector partners. ATP is designed to attract private firms as partners and to influence their behavior, even as it relies on the special knowledge that firms have of their relevant markets, and on their profit motive and their sharing of the costs to increase program efficiency. It is not surprising, then, that a number of evaluation studies have provided insights as to how and why firms engage with ATP to develop new technologies, how they are affected by that engagement, and the significance of the interactions to the firms. This section summarizes related findings on private firm effects drawn from 13 studies listed in Table 9–1. Besides author and publication date, the table’s column headings indicate the six major sub-themes to be covered. 295
Addressing a “Financing Gap”
Private firms face barriers to innovation that have a long development and commercialization time horizon. The Harvard-MIT study sought to develop a better understanding of “the decision-making process within firms, and within outside financing sources, as it relates to the funding of early-stage, high-risk technology projects...” 296 It took as a point of departure a key premise leading to the establishment of ATP:
The Harvard-MIT study looked to venture capitalists and others to help define the financing conditions faced by innovative firms. The views expressed by study participants “make clear the existence of a serious gap between the public resources available for academic and national laboratory research and the ability of private venture investors to finance research to reduce the new technical ideas to commercial form...the ‘Valley of Death’ in R&D...” 297
According to David Morgenthaler, a past chair of the National Venture Capital Association and a participant in the Harvard-MIT study, “A useful way of looking at technical risk is to assess at what stage of development of a technology an institutional venture capitalist should invest.” He provided as guidance the list below. 298 In his view, there is “little or no role for government in the later stages of development” but in the earlier stages of R&D, where venture capitalists generally do not want to invest, private firms need support by the government. Mogenthaler’s list of why venture capitalists should and should not invest helped to define where government support is needed: Venture capitalists should:
Morganthaler elaborated as follows:
In a related study conducted by Harvard University researchers, 299 it was found that:
The struggle of small private firms to find funding for the innovations they eventually brought to ATP has been documented in case studies. Gompers and Lerner interviewed seven small, innovative companies in the Boston area to find out what role “ATP funding has played in the company’s evolution.” 300 They found reasons each of the companies turned to ATP for funding, and concluded, “The Advanced Technology Program has substantially expanded and enhanced the R&D activities of our seven-company sample.” 301 They concluded that despite the growth in venture capital funds during the mid-1990s, less than one tenth of one percent of business startups annually received venture financing. 302
Servo used the experiences of several firms to describe how ATP funding fit within the landscape of potential funding sources. As recounted by one firm spokesperson in her study, “We sought ATP funding as a means to develop novel process technology which exceeded our existing financial capability and which we were unable to convince venture capitalists and angel investors to back due to high technical risk.” 303
ATP’s Status Reports also documented circumstances in which firms were unable to obtain private sector financing, and sought ATP assistance in order to accomplish their innovation. For example, in documenting the ability of the two young proprietors of Calimetrics, Inc., to find financing for their pit depth modulation (PDM) technology, the Status Report gave the following account: 304
Detailed case studies also shed light on the financing gap. According to Ehlen, who investigated the joint venture to develop flow-control machining technology, the end users of the technology tend not to fund the research of their suppliers and are unlikely to adopt a new process not yet proven to work; the suppliers lack the capital to do extensive in-house research—“particularly not high-risk research”; and university researchers lack the interest and resources to self-fund such a research program on their own. But with ATP funding, a supply-chain collaboration was formed to undertake the project. 305
Why do companies turn to ATP for funding? Several company spokespeople appearing at the National Research Council’s (NRC) workshops on ATP offered their own explanations. For example, David Ayares of PPL Therapeutics, Inc., a company working on xeno-organ transplantation, said: 306
Feldman and Kelley, in their survey study of a large group of 1998 ATP applicants, found that most companies turned down by ATP did not proceed with their R&D projects. If funding were readily available from other sources, it would seem that more of the companies would have been able to pursue their projects without ATP. 307 And, according to Powell and Lellock in their analysis of ATP Business Reporting System (BRS) data, ATP award recipients increased their own investments in their ATP-funded project as a result of obtaining ATP. 308 These findings provide evidence that ATP funding is complementary to, not a substitute for, private sources of R&D funds; that is, that it is filling a funding gap.
Influence on R&D Investment Choices
While some of the evaluation studies focused on the availability of funds and the timing of investment, others investigated the effects of ATP funding on firms’ R&D investment choices. Identified effects included: (1) increased innovativeness of research programs, (2) willingness to commit to longer-term research programs, and (3) ability to tackle larger scale and broader scope problems and opportunities.
Powell and Lellock, in their analysis of BRS data, found evidence of increased innovativeness attributed by firms to ATP. Figure 9–1, for example, indicates, “With the assistance of ATP funding, organizations are pursuing different R&D than they would have undertaken without ATP funding.” 309 Powell and Lellock found that, relative to what the project participants thought would have happened without ATP, 73% of participants in ATP projects “were more willing to accept technical risk” while 67% had “increased interest in performing longterm research” and 61% “had increased the R&D scope.” 310
Source: Business Progress Reports from 403 organizations in 198 ATP projects funded 1993–1997 — after one or more years of ATP funding.
BRS data also provided specific examples of estimates by project participants of ATP’s effect on their innovativeness. The participants identified the single most important parameter associated with technical success, and then specified the baseline metric, their project goal metric, and the expected value to be attained without ATP assistance, all in terms of the most critical parameter. This approach allowed comparisons. Table 9–2 provides a few examples drawn by Powell and Lellock from the BRS database. One of the examples is for gene sequencing. At the beginning of a project to develop advanced gene sequencing technology, the rate of sequencing was one gene per day. With ATP funding, the goal was to develop the ability to sequence 100 genes per day. Without the ATP award, the participants projected that they could only achieve a sequencing rate of five genes per day within the planned time frame. 311
Source: Drawn from Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, p. 11.
Feldman and Kelley provided additional evidence that ATP has influenced firms’ R&D investment choices. In their comparison of winners and non-winners, they found evidence that ATP encourages firms to pursue new technical areas, outside the scope of the firm’s past R&D activities. They found that among all applicants, 28% proposed in a technical area new to the company, but of award winners, 47% proposed in a new technical area. They concluded “these differences suggest that the ATP’s cost-sharing partnership with industry is indeed underwriting the efforts of firms...to initiate risky projects in new technical areas.” 312
Halo Effect from ATP Award
As indicated earlier in the report, the existence of a “halo effect” on firms receiving ATP funding has been a recurring theme among project participants. In ATP’s first survey, Solomon Associates found 100% of single-firm award recipients and 60% of joint venture participants reporting that their ATP award had “enhanced the organization’s credibility in some way which resulted in impacts ranging from enhancing the image of the company with potential investors to increased public awareness of the technology.” 313 This was mentioned by many of the respondents as “the single most important effect that the ATP award has had on your organization thus far....” 314
In ATP’s second major survey, which focused on participants in the first three ATP competitions, Silber & Associates replaced halo terminology with “increased credibility.” It concluded that 90% of participants benefited to a “great” or “moderate extent” from enhanced credibility associated with the award. 315 According to Silber’s report, “Credibility is almost equal in importance to the direct financial impact of the award, particularly since increased credibility can mean increased business activity.” 316
The on-going BRS survey of ATP project participants used the terms “increased credibility” and “halo effect” interchangeably. Powell and Lellock’s analysis of BRS survey data showed 93% of participants citing increased credibility due to their ATP award. 317 According to the researchers, “The ‘halo effect’ may be expected to be of particular benefit to ATP-funded small businesses, which have little if any market presence and typically very limited financial resources at the time of the ATP award.” 318
Among five major ATP effects Laidlaw identified as helping project participants, was a halo effect. 319 Laidlaw’s reporting of interviewee comments suggested the connection between the ability of firms to carry out high-risk research, as discussed in the previous section, and the halo effect:
Feldman and Kelley’s survey of 1998 winners and non-winners rigorously tested the hypothesis that ATP confers a halo effect. 320 To do this, they took into account other factors that could have explained improvement in the firms’ ability to attract funding from other sources. As stated in the NRC’s condensed version of the Feldman and Kelley study, 321
Acceleration of Technology
Because helping U.S. businesses accelerate development and commercialization of technology is a legislated mandate for ATP, a number of evaluation studies have investigated and measured this effect. The term “acceleration,” as used here, encompasses a wide scope of changes in timing, ranging from months to years, to cases where the technology would not have been developed at all without ATP. And, as noted by Laidlaw:
Surveys of project participants provided the earliest evidence that participating in ATP was allowing firms to speed up technology development and commercialization. In the first ATP survey, Solomon Associates found that 69% of participants had saved time in developing their technology—usually in the one to five year range—and 35% of these considered the time saved of critical importance. 322 Three years later, Silber & Associates found 95% of participants reporting a shortened R&D cycle because of ATP, with most reducing their R&D cycle by at least two years. 323 ATP’s BRS, which has regularly tracked acceleration, added questions about the participants’ window of opportunity to questions about the amount of time saved and the importance of saving time. In a further departure from the earlier surveys, the BRS looked at acceleration in terms of the various applications of the technology, making it not strictly comparable to the Solomon and Silber results.
Figure 9–2, reproduced from a BRS report, 324 shows the three aspects of timing tracked by this survey: (1) the estimated reduction in time to market, (2) importance of speed, and, to put time into perspective, (3) the perceived length of the window of opportunity. As Powell and Lellock observed:
Source: Business Progress Reports for 974 applications being pursued by 464 companies in 258
The Feldman-Kelley survey provided additional evidence of an acceleration effect of ATP on R&D. As shown by Table 9–3, winners proceeded with their proposed R&D, while less than two-fifths of the study’s control group of non-ATP-winners had started any aspect of the R&D project they proposed to ATP a year after being unsuccessful in the ATP competition. 325 The results implied that the winners also might not have started without ATP, but otherwise provided no specific estimates of time saved.
*Companies reporting more than one type of alliance are included twice.
Source: ATP’s Business Reporting System. Data for 747 applications being pursued by 356 companies in 198 ATP projects funded 1993–1997—after one or more years of ATP funding.
Laidlaw’s interview-based analysis of the acceleration effect produced a clearer understanding of how and why ATP caused acceleration to take place, and even placed rough values on it. In her interviews with ATP award recipients, she asked why saving time is important to the firms. She investigated how the firms actually saved time through participating in ATP and whether the effects extended throughout the firm, beyond the ATP-funded project. 326
Laidlaw found that 96% of interviewees assigned a rating of “very important” to reducing cycle time, in order to meet competition. 327 When asked how much time they had saved due to ATP participation, the median response was “by 50% or three years.” 328 Table 9–4, from the study, represents Laidlaw’s attempt to obtain quantitative estimates of the dollar value of a year of time saved. For the 54% of surveyed firms responding with quantitative estimates, “The median estimate of the economic value of reducing the applied research cycle time by just one year is $5 million to $6 million.” 329
Firms interviewed by Laidlaw identified five principal factors that helped them cut time. These are summarized in Table 9–5. The most frequently mentioned factor was “ATP’s requirement for an integrated business and R&D plan with its emphasis on concurrent engineering…” 330 Spokespeople mentioned the integrated planning requirement more often than ATP’s funding as an important factor in accelerating their technology development. Elaborating, firms spoke of the positive effect on their timing of having a well-laid-out plan and following the plan without interruption, the stability lent by ATP’s involvement, early initiation of work with potential customers and users, integrating the voice of the customer, using a systematic approach, developing definable timelines, benchmarking, and selecting potential technology applications early in the process, and enhancing documentation procedures.
But, if ATP’s funding effect is defined more broadly—taking into account both ATP’s direct funding plus the improved ability of firms to attract additional financial assistance from others through ATP’s “halo effect”—funding is the effect most frequently mentioned by firms as speeding their technology projects. 331
Laidlaw found that 86% of those interviewed expected the time saved in the R&D stage to flow through to later project stages. 332 Thus, speeding R&D typically speeded commercialization.
She found that 86% of firms found a carry-over of time improvements to their other technology development projects. Table 9–6 summarizes the responses, identifying ways the time reductions were transferred to other projects.
Source: Adapted from Laidlaw, Acceleration of Technology Development by the Advanced Technology Program, 1997, Table 10.
Developing enabling, generic technology—”technology platforms”—reportedly was a major factor allowing firms to speed other projects involving multiple applications of the ATP-funded technology. Applying the ATP-required integrated R&D and business planning to other projects reportedly also helped firms speed other projects. The ability to apply methodologies and processes developed in ATP projects to other projects was another factor identified as helping firms speed their other projects. Some firms further reported that they had developed a company culture emphasizing speed as a result of their ATP project participation. 333
Project case studies provided another look at ATP-induced acceleration of technology development. The RTI case studies, for example, estimated project acceleration ranging from “one to at least 10 years” for the seven tissue engineering projects included in the study. 334 In a more complicated case, the firms in the ATP-funded Printed Wiring Board joint venture reportedly would have delayed about half their research by at least one year without ATP’s assistance, and would not have undertaken the other half at all. 335
The set of mini-case studies for the completed projects provided yet another look at acceleration. Table 9–7 summarizes the acceleration effects of ATP funding for 44 of the first 50 completed projects, those for which data were available. 336 The majority of participants in completed projects reported they would not have proceeded at all without ATP funding. 337
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 24.
Increased Firm Productivity
ATP’s evaluation investigated program-induced changes in patenting by award recipients to gauge program impact on firm research productivity. Two of the evaluation studies examined—the Darby et al., study and the Sakakibara- Branstetter study—provided statistical evidence about how participation in ATP affected patenting behavior. 338
In determining how to best measure the impact of ATP participation, Darby et al., identified patents as “arguably the single best measure of commercial capture of advanced technology.” 339 They found this to be an appropriate measure because a “major purpose of ATP is to increase commercial capture of advanced technology.” 340 They produced an adjusted patent count measure by deflating the patent count of ATP participants using a factor for year-to-year changes in the average rate of patenting for all U.S. assignees of U.S. patents. 341 They estimated an increase in patenting averaging between five and 30 patents per firm per year of participation, attributed to ATP.Darby et al., concluded that the different levels of increase in patenting attributable to ATP participation are linked to differences in project structure. These project structure differences stem from whether the project is structured as a joint venture or single-firm project and whether universities are involved either as a joint venture member or as a subcontractor to either a single-firm project or a joint venture. They found that members of joint ventures experienced more of an increase in patents than single firms due to their ATP participation; and firms with university partners or subcontractors experienced a higher level of patent increase than those without university involvement. “Having all three— JV [joint venture], university partner, and a university subcontractor—appears to help the most.” 342
In addition to estimating an increase in firm patenting due to ATP participation, Darby et al., concluded that ATP-funded organizations accounted for an unusually high percentage of all U.S. patents granted to U.S. entities during the period 1988–1996. They did not attribute this fact to ATP, but rather suggested it signals “the very top, leading edge technology firms are getting involved [with ATP] as well as the top universities.” 343
The Sakakibara-Branstetter study also examined the impact of participating in ATP projects on the overall research productivity of firms that participated in ATP-sponsored joint ventures, again using patent data as a proxy for productivity. 344 Their study compared ATP participants with a control group over a common time period, rather than taking the before-and-after approach of the Darby et al., study.
Sakakibara and Branstetter’s findings suggested that participation in ATP raises the research productivity of participating firms. 345 They found that participating in ATP joint ventures increased patenting in the targeted technology areas above the level of patenting that participating firms showed prior to participating in the project. They estimated that, at the margin, each instance of participation in an ATP project raised a firm’s research productivity, as measured by patents, by 8% per year. 346 They found evidence that consortia in which participating firms are technologically proximate were systematically more successful than consortia in which participating firms are involved in diverse technologies. 347
Small Business Participation
Reflecting the attention given in ATP’s authorizing legislation to small businesses, ATP’s evaluation program has addressed the following question: “Have small firms been able to compete successfully against larger firms for ATP awards?” Powell analyzed BRS data to help answer the question. The data showed that the majority of ATP participating companies, including subcontractors, are small, and that 61% of awards have gone to projects led by small firms. 348Powell further investigated the performance of small companies who received awards, seeking to answer the following question: “Are there signs of commercial success and economic impact from small firms in ATP?” Powell concluded that small firms were showing strong progress toward early stagecommercialization. 349
Table 9–8 shows results from her comparison of small and larger firms in earning revenue, adopting process improvements, and filing for patents.
Source: Powell, Business Planning and Progress of Small Firms Engaged in Technology Development through the Advanced Technology Program, 1999, p. 45.
Commercial Progress, Company Growth, and Private Returns
“New technical knowledge must be used if economic benefits are going to accrue to the nation. This generally means the introduction into the market of a new product or process by the innovating firm, its collaborators, or other companies that acquire the knowledge.” 350
Innovating firms making progress toward commercialization may experience growth, increased capitalized value, higher sales, increased revenue, and return on investment. Their close collaborators and licensees are also positioned to make early commercial progress. The activities of the awardees, their collaborators, and licensees comprise ATP’s “direct path to impact.” 351
Table 9–9 summarizes the commercialization progress of the first 50 completed ATP projects. Examples of the early technologies and the products and processes developed from them are given in Table 9–10. 352 The individual project summaries in the Status Report provide accounts of each innovator’s specific progress toward taking their technology further into commercialization.
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 17.
Employment Gains Among Small Innovating Firms
Company growth, as indicated by employment increases, is easier to isolate among ATP participants by examining small, single-applicant company recipients of ATP funding rather than large, multi-divisional firms, joint ventures, or nonprofit organizations. Employment changes in larger companies, non-profits, and joint ventures are too complex to link directly to the ATP project. But for small companies, “Rapid growth is generally a signal that the small innovating company is on the path to taking its technology into the market.” 353 The Status Report provides an overview of employment changes experienced by small companies included in the first 50 completed projects. Figure 9–3 shows the distribution of small companies by their percentage change in employment.
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 18.
As the Status Report points out, “Sixty percent have at least doubled in size; four of them have grown more than 1000 percent.” 354
Returns to the Innovators
Returns to innovators can be measured in several ways. For the seven firms that carried out the tissue engineering projects—the technology focus of the RTI studies—the authors computed a composite measure of direct company returns. Although they calculated a measure of private returns for each project, to preserve proprietary information, they presented the results as a composite. The estimated composite internal rate of return (IRR) to the companies was 12%.
Consistent with ATP’s goal of generating broad-based benefits, the estimated composite private return was much lower than the composite social return of which it is a part. 355
Pelsoci’s case study of closed-cycle air refrigeration (CCAR) estimated revenues to private company innovators. 356 The study projected the present value of projected revenues from CCAR installations in the food service markets, volatile organic compound market, and liquid natural gas industries to total $65 million. (The IRR was not provided.)
Source: Extracted from Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, Appendix A, pp. 253–258.
Finally, Ehlen’s case study of flow-control machining technology estimated increases in annual sales attributable to the project, focusing on three groups: (1) the main technology developer, Extrude Hone Corporation; (2) the first-line users of the technology, a group of aluminum casters; and (3) the automakers who would purchase the affected engines and install them in vehicles. 357
As Table 9–11 illustrates, for this process technology, the developer’s change in annual sales, shown in the bottom row, was relatively small. Again, the findings are indicative that the benefits to the innovator are much less than estimated social benefits.
Source: Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999, p. 45.
One of ATP’s legislated mandates is to “aid industry-led United States joint research and development ventures...” Evaluation studies have variously focused on the structure and formation of new ventures, the role of universities in joint ventures, factors determining their success, their stability over time, and their benefits and costs. This section draws findings from 10 studies listed in Table 9–12. The discussion is organized along the lines of the sub-themes identified in the last five column headings of the table.
Collaborative Activity, New Venture Formation, and Attribution to ATPFindings of four surveys, plus the completed project Status Report provided measures of the frequency of collaborative activity among ATP projects. The results are summarized in Table 9–13.
The earliest survey of ATP participants by Solomon Associates, published in 1993, found that 46% of those companies interviewed after one year in the program had formed strategic alliances to further advance the technology associated with their ATP project. Some others were in the process of forming collaborative relationships, and others said it was “too early.” Only one company indicated no collaboration and no near-term plans to collaborate. 358
While the collaborative activities identified by Solomon were linked directly to ATP projects, the survey made no attempt at establishing a counterfactual comparison. Rather, it merely reported the frequency of collaboration associated with ATP projects, and captured some participants’ comments concerning the benefits.
Silber & Associates’ 1995 survey measured the frequency of collaborative activity, and also identified the nature of the collaborations, the number of partners, the degree of success, and the extent to which the relationships were new ones, “formed expressly to carry out the ATP project.” Silber concluded: 359
The Silber survey specifically investigated the extent of collaborative activities among single-company applicant projects, finding that 52% of the single company applicants had formed collaborations and that they brought an average of four outside companies into their projects as subcontractors. The study found that each joint venture project included an average of six members, and 43% of the joint venture members “forged subcontracting relationships with an average of five additional companies.” 360
Powell and Lellock, from their analysis of BRS data, provided further evidence that collaborative activities are extensive among ATP projects, and that ATP played an important role in stimulating the collaborative activities. Analyzing data from 415 participants in 198 ATP projects—including single-company applicants and joint ventures—they reported that of the 86% of respondents reporting collaboration, 69% reported that ATP “to a great extent” was responsible for bringing about the collaboration. 361 Powell and Lellock also found evidence of a substantial formation of strategic alliances aimed primarily at commercializing ATP-funded technologies. These included alliances with suppliers, customers, joint production, distributors, and licensing partners. 362
Feldman and Kelley’s comparisons of ATP award winners in 1998 with a control group of non-winners added to knowledge about the extent and importance of collaborative activity in ATP projects. The researchers asked winners and nonwinners alike about their partnering activities and whether their “most important collaborator” was someone with whom they had or had not partnered before. As indicated in Table 9–14, there was a high (79%) and equal tendency among winners and non-winners to propose projects entailing collaborative relationships. But the award winners were more likely to have a new collaborator as their most important research partner. Thus, of the 79% of winners whose proposals involved collaborative relationships, 59% were principally with organizations with whom they had not previously partnered, while for non-winners, 42% were with new partners. 363
Feldman and Kelley concluded that ATP successfully encouraged applicants to propose projects featuring collaboration, frequently with entirely new partners and, moreover, that ATP’s selection process advanced its goal of encouraging new venture formation by favoring the selection of proposals involving new partnerships. They provided the following pointed explanation of the importance of collaboration and the government’s role in making it happen: R&D collaboration is widely recognized as important to a firm’s R&D strategy for learning about technical advances in other organizations. But establishing new collaborative ties to other firms is difficult. An important role for government in public-private partnerships is to foster the formation of new R&D collaborations by reducing the costs to the firm of establishing these relationships. (p. 17)
Feldman and Kelley also tested whether the types of collaborative linkages differed among applicants to ATP who received and did not receive ATP funding. They found no statistically significant difference among winners and non-winners in their linkage to universities; that is, both groups were well linked to universities. But they did find that winners had more linkages to other businesses than non-winners. Based on their finding, Feldman and Kelley stated:
Source: Feldman and Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, p. 20.
While the previous table shows collaboration at the proposal stage, Table 9–15 shows for the first 50 completed ATP projects the extent of collaborative activity over the entire project life, and the different types of collaboration. In a group comprised of only 16% joint ventures, 84% of the projects reported collaborative relationships—some for R&D, some for commercialization, and some for both. These statistics further indicate the extensive collaborative activities found in single-company projects. 364
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p 4. 367 Ibid., p. 5.
University Representation and Roles in ATP Projects
ATP’s authorizing language focused on the needs of America’s firms and industries. Following this intent, ATP focused its early program design and selection criteria on variables and relationships deemed salient to industry, treating universities as supporting players. But observing the behavior of firms that applied for ATP awards, it became apparent that collaborative relationships between firms and universities were more important than suggested by the original program language. Over time, it became evident from the proposals submitted to ATP that firms were choosing to include universities as R&D collaborators in major ways.
Several different studies investigated university representation in ATP projects, and they all found evidence of substantial university involvement. ATP’s Status Report provided an account of university involvement in the project case studies, and a count of the number of completed projects with university relationships. It reported that 48% of completed projects had university collaborations. 365
Feldman and Kelley’s exploration of 12 different types of linkages between ATP projects and universities, 366 for example, found that the entire large sample of 1998 applicants surveyed had ties to universities. Feldman and Kelly reported that 57% of all ATP projects had universities as joint venture members or subcontractors as of 1999. More often universities were in a subcontractor role. 367 Analysis of the first 50 completed ATP projects also showed the “close R&D ties of firms with universities.” Nearly half the projects (48%) involved universities. 368
Other university-related studies focused primarily on their roles in ATP projects and were covered in more detail in Part II, Chapter 4. These included the Hall et al., study 369 that found projects with university involvement are likely to experience more difficulty and delay, but also are more likely not to be aborted prematurely; the Kogut-Gittelman study 370 that found that firms with weak in-house research capabilities boosted their capabilities by allying with university scientists; and the Liebeskind study 371 that found an inverted path of knowledge flow from industry to university.
Mid-Course Changes in Collaborative Relationships
In his case study of an ATP-funded joint venture, Link provided an up-close look at the formation and reformation of collaborators in the joint venture over a five-year project life. He showed that while the makeup of the collaborators changes over time, the roles remain relatively constant. 372 Table 9–16 illustrates membership changes in the Printed Wiring Board (PWB) research joint venture.
The Silber survey provided statistical information on mid-course changes in project participants. The survey found that 59% of projects were carried out without changes in their collaborating organizations. It also found for “23 percent of the projects, at least one participating company was changed to a different company, and [for] 18 percent, at least one participant, along with that company’s piece of the project, was dropped altogether.” 373
Such changes in collaborative arrangements are important because they give rise to an issue for ATP management: at what point does a change in project makeup or associated goals no longer comply with the original criteria by which the project was selected for an ATP award? 374 Analyzing project changes helps management understand this issue better and is, therefore, a valid component of evaluation.
Determinants of Collaborative Success
Dyer and Powell investigated factors increasing or decreasing the likelihood of success of collaborations. They interviewed company representatives and government project managers for 18 joint venture projects in the automobile industry, asking them a series of questions related to the success of their collaborative activities. They found that greater knowledge sharing and more effective coordination among participants characterized the more successful joint ventures. They developed hypotheses from the interviews about factors influencing the extent to which participants shared knowledge and influenced the costs of coordinating the venture’s activities, and expressed future plans to test their hypotheses through a survey.
The path diagram in Figure 9–4 shows the key factors participants said influenced the degree of success of their joint ventures, the direction and nature of impact, and the linkages from these factors to measures of outcome success. 375
Starting on the left, the first set of circles contains five factors that participants identified as influencing their success in collaborating. Moving to the far right, the highlighted circle lists the main measures used by the participants in measuring their success. The center of the figure shows the key mechanisms through which the five factors on the left influence the success measures on the right. These factors are trust, information sharing, and coordination costs. The plus and minus signs along the directional lines show the nature of impact in the direction indicated. For example, geographic proximity of the members is shown to contribute to success in two ways: a positive effect on information sharing contributes positively to success; a negative (reducing) effect on coordination costs also contributes positively to success because the costs are negatively associated with success.
Source: Dyer and Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, 2001, p. 11.
Source: Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. 16.
According to participants in the Dyer-Powell study, ATP contributed to their joint venture success by (1) accelerating the development of high risk technologies, (2) increasing project stability, (3) getting projects through particularly difficult periods in their life cycles, (4) overcoming barriers to collaboration, and (5) increasing upfront planning.
Sakakibara and Branstetter also touched on the question of what determines consortium success in their research. 376 Looking at the impact of participation of companies in consortia on their patenting rates, the researchers stated:
The structure of collaborative arrangement is another factor important to the collaboration and the project’s overall success. Sketching a project’s collaborative structure can help project selection boards analyze key roles, responsibilities, and linkages among collaborators, and may help identify missing elements that threaten future project success. Ruegg depicted a variety of collaborative arrangements that have been used by ATP project participants, two of which appear in Figure 9–5. 377 The first shows a vertically structured joint venture and the second a hub and spoke structure with an important member missing, which likely adversely affected project success.
Assessing the Benefits of Collaboration
The 1992–2000 surveys of ATP participants investigated the perceived benefits of collaborating. The earliest survey conducted by Solomon Associates found that 80% of joint venture participants “discussed a myriad of benefits they had experienced to date because of the collaboration...” 378 But the study captured the nature of the benefits only in several anecdotal comments, such as:
Source: Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, 1999, pp. 15–16.
Silber & Associates took the investigation of collaborative benefits further, compiling statistics on the extent to which companies benefited from the collaborative relationships of their ATP projects. This survey found that 60% of respondents benefited “to a great extent” from the collaboration, and 35% “to a moderate extent.” 379 The survey also tallied the frequency with which respondents said they experienced specified collaborative benefits—such as to stimulate creative thinking—to a “great, moderate, or small extent, or not at all.” 380
Powell and Lellock, in the BRS survey, continued to probe the benefits of collaboration. They largely repeated Silber’s probe of specific benefits from collaboration. Table 9–17 below compares results of the two studies’ findings on specific collaborative benefits. The first column lists the identified beneficial effects of collaboration in declining order of importance according to the BRS survey. The second column gives the frequency with which BRS respondents said each benefit was “significantly enabled by collaboration.” The third column gives the frequency with which the Silber survey respondents said they experienced the specified benefit “to a great extent from collaborating.” 381There is a striking agreement that “stimulating creative thinking” was the most important benefit of ATP R&D collaborations. This finding is consistent with other findings, such as by Feldman and Kelley, that ATP caused firms to undertake new, challenging areas of research.
Assessing the Costs of Collaboration
The same surveys that investigated benefits of collaboration also probed the costs. The earliest survey, by Solomon Associates, looked only at joint venture participants in the first competition. It found that after their first year, 60% of the joint venture participants identified one or more problems from the collaboration.
Almost all the problems were related to administrative issues, specifically auditing and accounting practices and legal arrangements regarding intellectual property, rather than with the research collaborations per se . 382
With a greater understanding of the extent to which single applicants were collaborating with others in ATP projects, Silber & Associates investigated collaboration problems across all project participants—not just joint ventures. The study identified a wider variety of problems than did the Solomon study, but the study did not identify the frequency with which the different kinds of problems were experienced. 383
*The Silber survey’s closest matching category was called “increased customer acceptance.”
**The Silber survey’s closest matching category, for which the percentage applies, was called, “enabled you to develop technology while you engineered for volume manufacturing.”
The types of collaboration problems identified by the Silber & Associates survey were:
Some of the single applicants with collaborative arrangements implied that there are costs associated with formal joint venture collaborations by extolling the comparative advantages of their more informal arrangements. One cited the “single, clean decision-making chain...” the “better focus, better control, and increased flexibility with Single Applicant programs.” 384 In effect, they did not identify collaboration per se with increased costs, but rather with the jointventure form of collaboration. The freedom to apply to ATP as single applicants and use informal collaborative relationships to carry out proposed projects appears to be valued by firms.The survey conducted by Silber & Associates provided a breakdown of survey responses regarding the effectiveness of relationship with collaborators by year of award. Only 21% of the first award group—a group with an unusually high percentage of joint ventures—reported that their collaborations worked “extremely well,” while 55% of the second year’s award winners had collaborations working “extremely well,” and 67% of the third year’s award winners had collaborations working “extremely well.” 385 These results may imply a reduction in the administrative costs of collaboration that were identified early on; they may reflect a larger percentage of informal collaborations among the later groups; or other effects which caused the collaborations of the later groups to work better than the earlier ones.
Analysis of BRS data also provided evidence about the type and extent of costs associated with collaboration. Figure 9–6 shows the percentage of project participants who experienced (1) delays in beginning their R&D, (2) increasing project coordination and management costs, and (3) delays in product entry into the marketplace. Note that only 9% reported that their costs were increased “significantly.” 386
Source: ATP Business Progress Reports for 974 applications being pursued by 463 companies in 256 ATP projects funded 1993–1997.
Both the Silber survey and the Powell and Lellock analyses of BRS data have suggested that costs associated with collaboration do enter into the decisions of ATP awardees but both have concluded that costs have not been widespread or serious overall in ATP. 387 The Silber study concluded, “For the majority of participants, however, the problems have been only minor stumbling blocks, resolved without serious consequences.” 388 Additionally, the study noted, “more than 90 percent of all participants said their collaborative relationships have worked out ‘extremely well’ or ‘fairly well.’” 389
Silber & Associates noted that 92% of respondents “said their ATP experience has piqued their interest in working collaboratively with other companies, and 96 percent of joint venture participants reported that this experience would influence them to engage in another joint venture down the road...” 390 This conclusion further indicates that the problems with collaboration in ATP projects have not been very serious.
The concept of economic spillovers has occupied a central place in the case for a public sector program like ATP, and has helped shape many of ATP’s program design features. Jaffe identified finding evidence of larger-than-average spillovers as a requirement for program success. Proof of large spillovers counters charges of corporate welfare, and supports public investment. Accordingly, a considerable portion of ATP’s evaluation program has been directed at explaining the concept of spillovers, and empirically estimating its presence and magnitude. This section draws material from the studies listed in Table 9–18. As may be seen from the table, nearly all the listed studies addressed either market or knowledge spillovers; only the first two addressed spillovers more broadly defined.
*Denotes unpublished, draft reports.
Early Planning to Capture Spillovers
Mansfield’s pilot study for ATP, conducted in the early years of the program, explored the applicability of his model for estimating private and social rates of return from the commercialization of new products and processes by firms receiving ATP support. 391
Mansfield’s study also highlighted the tasks that lay before ATP in attempting to adapt his model. As explained in greater detail in Part II, Chapter 4, Mansfield’s earlier applications of his approach used historical data, whereas ATP technologies were at best in the early stages of commercialization. His applications involved single products whereas ATP wanted to apply his model to technology platforms generating multiple products. Perhaps most importantly, his model sought to capture only market spillovers, whereas ATP was also interested in estimating knowledge and network spillovers.
Jaffe’s construction of a market and knowledge spillovers framework broadened the Mansfield social benefit framework. 392 As earlier illustrated in Table 4–1, Jaffe identified three different sources of spillovers of relevance to ATP: knowledge spillovers, market spillovers, and network spillovers. In effect, ATP sought to increase and measure each of these forms of spillovers.
ATP’s evaluation program has focused on the first two, market and knowledge spillovers, and, at the time of this study, had not yet estimated network spillovers. As defined earlier in Table 4–1, market spillovers occur when market dynamics cause some of the benefit of a product or process to flow to market participants other than the innovating firm, and knowledge spillovers occur when knowledge created by one agent is used by another without full compensation.
Estimating Knowledge Spillovers
The data revealed by the BRS and the completed project status report strongly indicate that as a portfolio the funded projects are generating outputs that potentially will lead to both market and knowledge spillovers. 393 These data document publications, patents, patent citations, collaborative linkages, and products and processes—all from which spillovers can result.
Feldman and Kelley found that ATP is selecting projects with attributes conducive to generating large knowledge spillover effects. These attributes include linkages of proposing firms to other organizations, and positive attitudes of award winners toward information sharing and knowledge transfers to other firms. As Feldman and Kelley pointed out, “The more embedded a firm is in a network of such inter-organization ties, the more quickly the knowledge generated by the firm is expected to be absorbed by other organizations in the system.” 394 They found “...on average, award winners have a more extensive set of ties to other businesses than do non-winning applicants.” 395 They also found:
Of the several evaluation studies that addressed aspects of spillovers, the Fogarty et al., study most directly focused on measuring knowledge spillovers. 396 This study was also notable for its methodological ingenuity. As explained in the treatment of the model in Chapter 8, the researchers applied systems analysis and fuzzy logic to analyze knowledge spillovers within networks of organizations.
To review, the method uses patent citations as a proxy for flows of scientific and technical knowledge. It uses citations to publications and earlier patents recorded on patent awards to construct networks of communication and influence among organizations within an innovation system. The direction and strength of relationships are determined by patterns of citing and sourcing among organizations.
One contribution of this study is that it constructs “networks” of organizations that extend beyond single interactions. The fuzzy logic and systems analysis components of the model are used to estimate the strength of the relationships among organizations. The model provides a far more detailed and comprehensive mapping of interactions of organizations important for the generation of knowledge spillovers than is possible in studying only interactions among members of a joint venture. For ATP projects generating pre-competitive, enabling technologies, the researchers identified the characteristics of R&D networks summarized in Table 9–19.
Although presented as an exploratory study with several of its key components yet to be fully tested, Fogarty et al., demonstrated use of the technique in identifying the knowledge spillover capability of an optical recording project funded by ATP. They showed that the project had an inherently high spillover potential and, hence, was a suitable selection choice for ATP.
Estimation of the magnitude and composition of knowledge spillovers also was a central objective of the Cohen and Walsh study. 397 This study advanced analysis of spillovers by its explicit attempt to link spillovers to appropriability; that is, the degree to which firms are able to protect the profitability of their own inventions and the strategies they use to appropriate the economic benefits from these inventions. The researchers noted that appropriability is affected by the mechanisms (e.g., patents, trade secrets) that firms use to appropriate economic rents 398 attributable to R&D, and that appropriability is a double-edged sword. On the one hand, a firm’s inability to fully appropriate the benefits of its own R&D reduces its incentives to invest in R&D; on the other hand, the inability of rival firms to fully appropriate the benefits of their own R&D constitutes a spillover to the first firm.
To review, Cohen and Walsh used data from the Carnegie Mellon survey to construct measures of appropriability, and then linked these data to other measures of intra-industry information flows, and firm and industry economic variables. Controlling for several variables, they found that intra-industry information flows complement firms’ own R&D efforts. The finding is consistent with the core propositions that led to ATP’s establishment and its key design features. In particular, by selecting generic technologies applicable to many firms both upstream and downstream and by supporting appropriate joint ventures, ATP can foster the generation of knowledge spillovers, and thus increases the productivity of a firm’s R&D.
A variety of studies provide additional accumulating evidence of the potential of projects for large knowledge spillovers. The completed project status report, for instance, included patent citation trees showing how knowledge generated by ATP was taken up by other organizations. Figure 8–5 in Chapter 8 shows an illustrative patent tree for a patent from an ATP-funded project.
Documentation of extensive collaborative activities, reported earlier in this chapter, showed the establishment of linkages across organizations and persons through which information potentially can flow, generating knowledge spillovers. Fogarty et al., applied their new model to an ATP-funded optical recording joint venture, illustrating the approach and providing a more rigorous way of assessing these linkages.
Estimating Market Spillovers
Several ATP-funded evaluation studies have sought to estimate the magnitude of market spillovers of ATP projects. In one of these, a study by Research Triangle Institute (RTI), researchers imputed a measure of market spillovers for a portfolio of seven ATP-funded products in medical technology, as the gap between estimated social and private returns. 399 The modeling approach was presented in Chapter 6.
Table 9–20 shows the RTI study’s estimated composite social returns and private returns on investment for the seven projects examined in the study. Row (1) data are for the project as a whole. Row (2) data are based on just that part of the project’s social return attributable directly to ATP and ATP’s share of investment costs. Row (3) data apply to the project awardees. Row (4) data are the difference between (1) and (3). Row (5) data are the difference between (2) and (3). The market spillovers—the gap between social and private returns—are quite large. As might be expected, spillover estimates were particularly large for those cases where RTI researchers were able to estimate the value of changes in quality adjusted life years for patients from the new and improved medical treatments, in addition to treatment cost differences.
Source: From Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, Tables 1–3 and 1–5
Based on these results, the authors concluded:
Another study that estimated market spillovers is Pelsoci’s study of CCAR technology, estimated as the difference between social and private returns. 400 After working through the technical performance features of the new technology and identifying pathways to market, the study estimated social and private returns based upon a conservative base case and an alternative optimal scenario. Like the RTI study, the Pelsoci study found a large gap between the estimated social and private, suggesting a large spillover effect. The estimates are for market spillovers only, because the estimated benefits are derived from sales of CCAR units.
Austin and Macauley used their cost index approach to estimate market spillovers from new technologies. 401 Applied to the development of two digital data storage technologies funded in part by ATP, the model yielded an estimate for consumer welfare gains over 5 years of $1.5 billion and $2.2 billion, respectively, for each of the two technologies.
State and International Technology Programs
Prior to the establishment of ATP in the United States, several industrialized countries already had in place similar technology development programs. Within the United States, the legislative creation of ATP in 1988 followed the establishment of nominally related programs by several states, and came near the peak of an approximately 10-year period when technology development programs were adopted by a number of states. The experiences of other nations’ public-private partnership programs and those of the American states were of keen interest to ATP. Program design features, operational issues, and program evaluation are among topics of common interest. In addition, in the case of state governments, ATP has sought to identify ways in which its specific features could be effectively integrated with those of state governments.
From the five studies listed in Table 9–21, this chapter draws findings on how ATP compares with programs of other countries, and how it relates to state run programs.
Comparing ATP with state programs and programs abroad fulfils several specific objectives, including: (1) coordination of federal and state activities, (2) distillation of best practices from comparable programs, (3) compliance with legislative requirements on international reciprocity and national treatment, and (4) provision of a test bed for new evaluation models by drawing data appropriately from programs with a longer history.
Programs in Other Countries
There were two objectives in ATP’s examination of counterpart programs in other countries: (1) to learn from them and make ATP a better program, and (2) to implement a statutory requirement that ATP consider proposals from U.S.-based subsidiaries of foreign-owned companies only if they passed three tests beyond those applied to proposals for U.S.-based firms. One of these tests essentially requires national treatment, namely that the parent company is incorporated in a country that affords U.S.-owned companies the opportunity to participate in government-funded joint ventures comparable to those of any other company in that country. Enforcing this provision required ATP to keep abreast of counterpart programs in other nations.
Chang’s framework and lexicon for identifying the key program features of technology development programs provided the structure to accomplish both of the above objectives. 402 She identified analogues of ATP “in many countries of varying sizes and economic maturity.” 403 Illustrated in Table 9–22 is her comparison of ATP with similar programs in Canada, the European Union, Finland, Japan, and the United Kingdom. The basis of comparison is their year of launch, mission, technical scope, eligibility regarding project lead, stage of research, level of formality of selection process, and cost-share requirement.
Source: Chang, A New Lexicon and Framework for Analyzing the Internal Structures of the U.S. Advanced Technology Program and its Analogues Around the World, 1998.
The table compares ATP with similar programs in Canada, Finland, Japan, United Kingdom, and the European Union in terms of the seven features listed.
Programs in Japan and Israel have figured more strongly in ATP’s evaluation than others. This is because several studies—those by Sakakibara and Branstetter and Griliches et al.—looked to these two countries for datasets that covered more years of operation than ATP. They used the data to provide a test bed for assessing the feasibility of specific evaluation methods for which adequate data from ATP was lacking at the time. In addition, these comparative studies also served to condition expectations about the timing and direction of project impacts.
Sakakibara and Branstetter used patent data from Japan to examine the long-term economic impacts of consortia-based joint ventures. 404 Having long-term data is essential to determining the impacts of consortia arrangements, because, as their findings indicate:
In a similar vein, Griliches et al., drew upon Israel’s over twenty years of experience with ATP-type programs to address central issues surrounding ATP’s establishment; that is, whether government support of private sector R&D served as a substitute for or complement to private sector R&D, and how it affected productivity. 405 The study served both to highlight the strengths and limitations of econometric modeling, for it performed quantitative tests of a salient policy issue while indicating the sensitivity of findings to model specification and sample selection. Their specific finding was that government-supported R&D generated high rates of firm productivity increases, but also noted that this conclusion “should be treated cautiously” because of potential problems in model specification, exclusion of relevant variables, and selection bias.
Relationship between ATP and State-Sponsored Programs
The essentially contemporaneous development of ATP and state technology development programs created both opportunities and complexities in coordination. A key question in the early years of the program was the character of the relationship between ATP and the state programs. Unlike the Manufacturing Extension Partnership Program, also established in the Omnibus Trade and Competitiveness Act of 1988 and located at NIST, there were few formal linkages between ATP and the state programs.
A two-volume ATP report, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, examined the character and potential of ATP and state program linkages. Volume 1 of the study, authored by Schachtel and Feldman, analyzed the structure of state programs, describing the range of activities supported by these programs, and indirectly pointing to the types of market failures that these programs were intended to offset. 406 Drawing on a model of product commercialization development by Goldsmith, the study illustrated various combinations of private-sector and public-sector relationships.
Analysis of the inventory of private-sector and public-sector relationships led to the following conclusions: State technology programs spanned the research and development continuum, ranging from an emphasis on basic research and human capital development, to pre-competitive research, to spin-off firms and product development. Some of the larger state programs provided support for each of these activities. In the main, though, state programs clustered around the downstream applied/commercialization segment, rather than the upstream research segment of the continuum. ATP, in contrast, centered its activities on technical challenges, supporting work primarily in the concept and development phases. The comparison of ATP’s focus with that of state programs is illustrated in Figure 9–7, using a version of Goldsmith’s nine-cell model that Schachtel and Feldman used in Table 4–5 to categorize state programs.
Source: Adapted by Ruegg and Feller from Goldsmith.
The Schachtel-Feldman study also highlighted the possibilities of firms combining support from both ATP and state governments, drawing upon ATP support, as above, for concept development, and then coupling this support with state support for downstream scaling up and product development support.
Volume 2 of the study, authored by Feldman, Kelley, Schaff, and Farkas, provided four case histories of the evolution of “technology-pioneering” firms that received support from ATP as well as state programs. 407 Of interest is the firm’s perspective of identifying and applying for various types of support from multiple sponsors. Although too limited in number to permit generalizations, the four cases do highlight several ways ATP and state programs augment each other. Consistent with the analytical framework noted above, ATP (and other federal agencies) were found to play the largest direct role in funding the upstream, concept development R&D, while the state programs were found to fund more of the downstream commercialization work. In the words of the researchers:
Moreover, the cases illustrate how state and federal government resources complement one another. (p. vi)
Measures of Overall ATP Performance
Empirically derived, comprehensive measures of a program’s ultimate outcome are a holy grail of evaluation. But, in the case of a young program like ATP where the long-term outcome is still unfolding, indicators of progress, partial measures of benefit, and assessments of effectiveness to date are reasonable, and even ambitious, evaluative accomplishments.
While some of the evaluation studies of ATP over the past decade focused on single issues, others attempted to provide broader assessments of the program. This section draws findings from the 13 studies listed in Table 9–23 that bear on the overall performance of ATP—its impact on national industrial competitiveness and the national capacity to innovate, its ability to deal appropriately with failed projects, and measures of portfolio performance, social returns and overall program effectiveness.
ATP’s Contribution to Project Impacts
It is not enough to know that the projects cost-shared by ATP have the desired impacts. To evaluate ATP’s contribution, evaluators must determine to what extent ATP caused these impacts. Did federal funds leverage or substitute for private funds? Might participating organizations have abandoned, delayed, or reduced the scale or scope of their efforts, or would they have taken the same steps without ATP? Since it is not unusual for a program to take full credit for any benefits that appear to emanate from it, the attribution requirement for ATP sets a comparatively high assessment standard in the sphere of applied evaluation.
Most ATP evaluation studies examined in this report have attempted to move beyond documentation of positive outcomes of one form or another associated with ATP funding to more formal tests of attribution and causation. The main techniques have been counterfactual analyses and control groups. These techniques and their application in ATP evaluation studies were discussed in Chapter 4. An earlier section in this chapter summarized evidence from studies showing that ATP has encouraged and made it possible for businesses to take on projects with higher technical challenges, start them sooner, carry them out faster, and enlarge their scope. In addition, another summary was provided of evidence that ATP has stimulated new collaborative activity. The evidence cited in these two earlier sections provides a backdrop for examining the broader measures of performance that have been developed for ATP and its portfolio of projects.
Improving the Industrial Competitiveness of the United StatesOne of ATP’s mandates is to improve “the competitive position of the United States and its businesses...” This requirement reflects at least in part the underlying concern at the time of ATP’s formation that U.S. businesses were often failing to commercialize technologies conceptualized in the United States.
By partnering with U.S. businesses and encouraging their long-term followthrough with new technologies, ATP seeks directly to affect the likelihood that U.S. companies will pursue the funded innovations. By funding broadly enabling technologies and promoting the dissemination of knowledge created, ATP further seeks to affect the competitive capabilities of U.S. companies.
While there are no aggregate, program-wide measures of ATP’s success in improving the competitiveness of the nation, there is a growing mosaic of evidence. The following study findings relate to ATP’s effect on competitiveness.
The Overall Competitive Position of Project Participants
The first insight to the program’s effects on competitiveness came from the first survey of project participants, conducted by Solomon Associates. 408
Three years later, the percentage of participants stating that their competitive standing had improved since they won the ATP award was 72% versus the 23% found earlier. Of the 72%, nearly half attributed the improvement to their ATP award “to a great extent” and another 28% “to a moderate extent.” Some of the nearly 30% who reportedly had not yet experienced any change in their competitive standing still hoped to: “We have high hopes the technology will increase competitive standing—potentially the ATP award will have a significant impact.” 409
The most recent findings on competitive position comes from the analysis by Powell and Lellock of BRS data, 410 but the data are not strictly comparable with the earlier survey data. The BRS approached the question by collecting data by application area rather than by project, such that a project with two application areas might experience gains in competitive standings in the one and losses in the other. An analysis of the BRS data, approximately seven years after Silber’s survey, showed 32% of participants reporting that their overall competitive standing had improved, and 9% that it had worsened, with respect to all applications of their ATP-funded projects currently on the market or being planned.
The earlier surveys may have captured primarily the views of companies in the pre-commercialization phase when they were receiving greater recognition and higher standing in their industry due to their ATP awards, whereas the more recent BRS survey may better capture the realities of the marketplace. As stated by Powell and Lellock 411 in the later data analysis:
An Industry’s Competitive Position
Link presented evidence that ATP affected change in the entire printed wiring board industry. 412 This finding is likely more significant in terms of evidence that ATP is meeting its legislated goal than the previously presented finding that ATP affected the competitive position of a certain percentage of project participants. The special significance of industry-wide competitive gains is that they more clearly represent a gain for U.S. industry in international competition.ATP’s printed wiring board (PWB) joint venture was primarily a horizontal collaboration among companies with the capability to mount advanced research in the PWB field. Members were not head-to-head rivals. They had substantial ties with other companies in the industry. The joint venture tackled a host of infrastructure problems that were reportedly causing U.S. suppliers of PWBs to fall behind foreign suppliers. The hundreds of small suppliers, largely lacking in advanced research capability, were going out of business at a sharp rate in the period prior to the project. Furthermore, according to Link, “The 1991 Council on Competitiveness report characterized the U.S. PWB industry as ‘Losing Badly or Lost.’” 413
Members of the joint venture reported that as a result of the project, “the domestic PWB industry as a whole has increased its competitive position in selected world markets...” 414 They also reported:
According to Link, at a meeting near the end of the project, the president of the National Center for Manufacturing Sciences “credited the project with saving the PWB industry in the U.S. with its approximately 200,000 jobs.” 415
Increasing the National Capacity to Innovate
A number of the evaluation findings suggest that ATP is adding to the nation’s capacity to mount future innovative activity. That is, the funded projects appear to build infrastructure important for innovation, and it remains after the funded projects are complete. This infrastructure consists of strengthened companies with more advanced capabilities and with more scientific and technical personnel with new knowledge and experience. It exists in new linkages among companies, universities, government laboratories, state agencies, and other organizations. It is evident in the newly created knowledge residing in people, journals, books, lab notes, and patents; and embodied in new products and processes. This infrastructure is critical to the nation’s capacity for further innovation.
The study by Darby et al., which investigated “whether ATP projects have a general effect on formation of new intellectual property within the firm,” 416 suggested that ATP is adding to the nation’s infrastructure important to future innovation:
A nation’s capacity to organize resources efficiently and to mount major new research activities effectively is a national resource of incalculable value. If, as the studies suggest, ATP is contributing to the nation’s capacity to innovate, this effect could be the single most important impact of ATP.
Dealing with Failed Projects
It is clear from the body of evaluation studies examined that not all ATP projects are successful—nor might they be expected to be if ATP is meeting its mandate of funding high-risk research. While failure in some projects should be expected, the rate of failure and how the program deals with project failure are questions that bear on program management and performance. A comprehensive failure analysis has not yet been undertaken for ATP at the time of this study, but several studies shed light on the issue of project failure and how ATP handled it.
In theory, projects fall on a continuum ranging from full success to complete failure, but in practice, few projects fit either extreme. Most achieve varying degrees of success, with something generally being learned even from projects that fail to start. As noted in the completed projects status report, which contains a section on terminated projects: 417
It is apparent from the body of evaluation studies that ATP does not simply declare all projects successful because they deliver some minimal level of knowledge. That said, projects that never start, that are terminated before completion, or that show no or few outputs are generally deemed failures.
According to the completed projects status report, approximately 5–6% of all ATP projects funded over the program’s first decade were terminated after the award announcement and before completion. 418 Figure 9–8 shows the reasons for termination. 419 Of the 16 first terminated projects, five were joint ventures whose members failed to reach agreement among themselves. Because the participants were unable to form their intended collaboration, ATP was unable to form the intended partnership with the joint venture, and, therefore, the projects never got off the ground. Eleven other projects were stopped prior to their completion. No ATP funding was awarded to those projects that never started, and only partial funding went to those that started but were terminated, the percentage depending on how long the projects ran.
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, p. 260.
Other identifiable poor performers are those receiving the lowest ratings of the Composite Performance Rating System (CPRS), discussed in Chapter 8. Based on CPRS ratings, nearly a quarter of the first 50 completed ATP projects were poor performers. 420
To be sure, the CPRS is a rough and approximate rating metric, but its use and the results show that ATP discriminates among its projects in terms of rating their level of success, and has a management tool for identifying the relative progress of completed projects in progressing toward program goals.
Measures of Progress, Social Benefits, and Overall Program Effectiveness
The body of evaluation studies completed over the past decade includes specific measures of the social impact of individual projects and, taken as a group, supports the formulation of general conclusions about the success of ATP. This section looks at studies that cut across multiple themes and program goals to suggest how ATP has performed at large.
Distribution of Projects by Overall Progress Ratings
The extensive data on outputs and outcomes have been analyzed and presented piecemeal to indicate project progress toward achieving particular goals. But, as noted, the data have also been combined and used to rate project performance in the overall achievement of ATP’s goals. Figure 9–9 shows the distribution of the first 50 completed projects by their overall CPRS performance scores. Figure 9–10 further shows the distribution of projects by their overall performance scores broken into five technology areas.
Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, p. 20.
Although they comprise only about 10% of total projects funded by ATP during its first decade, the first 50 completed projects provide a snapshot of portfolio performance free of project selection bias. As might be expected, the top performers, signaled by 4 stars, were limited to a relatively small group—16% of the total. Another 26% were in the next 3-star tier, also representing robust performance. Again, as might be expected, the largest group, 34%, were in the 2-star tier, exhibiting a moderate level of progress overall. As was noted in the previous section, the bottom performers comprised 24% of the total. 421
Note: When divided into technology areas, the numbers of projects in the sample in each area are too small to support drawing general conclusions about the performance of ATP projects by technology area. Source: Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 21.
Measures of Social Benefits
As ATP’s first decade drew to a close, evidence began to mount that “Estimated benefits attributed to ATP from just a few of the top-performing 50 projects ... far exceed the total ATP costs for all of the 522 projects funded to date [through 2000].” 422 This conclusion was based on estimated social benefits drawn from a collection of detailed case studies, together with implied benefits based on the less detailed “status reports” performed for the completed projects. Table 9–24 shows a compilation of measures of social benefits drawn from case study. The measures are partial, in that they do not cover all expected applications of the technology and do not include an estimate of the economic value of knowledge spillovers.
Additionally, the approach by which the estimates are derived is similar, in that they are based on benefit-cost methodology, but different in terms of their varying assumptions, time periods covered, base year in which dollars are stated, and specific modeling techniques.
The largest share of estimated social benefits supporting the above conclusion comes from RTI’s projections for three top performing projects that developed medical technologies. Benefit estimates included $98 million, $47 million, and $15 billion, from biocompatible polymers for cartilage repair, stem cell replication, and biomaterials for prostheses, respectively, all attributed to ATP. 423 The set of case studies provided evidence that ATP is funding a portfolio of projects whose benefits extend well beyond the direct award recipients. A major limitation was the prospective nature of the case studies.
Some of the detailed case studies did not so clearly separate out what are estimated private returns and what are social returns. Link, for example, used estimated R&D costs savings as a minimum estimate of project benefits. He estimated that the printed wiring board joint venture saved a minimum of $35.5 million in R&D costs, but did not attempt to apportion these estimated cost savings (proxy for benefits) between the awardees and other potential beneficiaries. 424
Beyond the estimated impact of the first 50 completed ATP projects, Ehlen’s study estimated an annual increase in GDP from the ATP-funded flow-control machining technology ranging from $142 million to $1.9 billion, depending on the assumed implementation path. He estimated that annual federal tax revenue would increase by between $34 million and $527 million. Pelsoci expected net benefits to result from “improved food safety, higher food processing yields and production rates, improved quality of processed foods, reduced harmful environmental emissions, additional U.S. exports, and crossindustry knowledge diffusion about ATP-funded innovations.” 425 He estimated net present value economic benefits from the ATP-funded closed-cycle air refrigeration technology between $459 million for a conservative case and $585 million for a more optimistic case.
The status report case studies for the stronger group of completed projects also provided evidence of growing revenue and potential benefits. For example, use of scalable parallel programming technology developed by Torrent Systems was reported to have allowed an early user to increase its revenue by $100 million annually. 426 Although these shorter cases do not provide quantitative estimates of net social benefits, except as they are drawn from previous in-depth economic case studies, they point toward future net benefits.
Overall Program Effectiveness
The NRC has provided the most comprehensive assessment of ATP to date. Drawing on expert judgment, informed by a large body of studies about ATP, and further informed by its broader investigation of federal public-private partnership programs, the NRC committee responsible for the ATP assessment concluded: 427
Addressing the issue of ATP’s success in advancing important societal goals, the NRC committee concluded:
Addressing the issue of whether ATP’s project selection process helps it to avoid displacing private funding sources, the NRC committee concluded:
The NRC committee also examined ATP’s evaluation program, concluding:
With respect to its review of the extant body of evaluation studies, the NRC committee concluded:
In addition to its findings, the NRC study made recommendations for “a series of operational improvements designed to make the program more effective.” Among the committee’s recommendations were two that relate to evaluation; namely, that outside assessments be released early to the research community, and existing efforts be enhanced to integrate assessment results into the ATP decision process. 428
Summary of Crosscutting FindingsIn sum, 429 findings on ATP’s performance and accomplishment, as evaluated and assessed from multiple methodological perspectives and by a large number of independent researchers and nationally regarded institutions, indicates that the program is achieving its overarching objectives of contributing to increased and accelerated rates of technological innovation leading to broadly enabling technology platforms, commercialization by U.S. companies, and thus improvements in the overall international economic competitiveness of the U.S. and its companies, and the generation of broadly distributed economic benefits.
Study evidence also points to the achievement of the specific sub objectives prescribed in its enabling legislation. ATP’s fostering of collaboration has been documented by survey, case study, and econometric studies. ATP’s attention to small businesses, and the success of small businesses in the program are supported by substantial study evidence. Study documentation of publications, papers, patents, citations by others, extensive collaborative relationships, and awards by third parties for technical contributions of projects all point to ATP’s success in stimulating the generation and dissemination of new scientific and technical knowledge.
Findings about spillovers are in line with theoretical requirements for the program—that they are ample and much in excess of private returns. Given that the study focus over the first decade was on measuring the value of market spillovers, the total stands to be much greater when estimates of knowledge spillovers are added to the market spillovers.
Finally, the extensive use of counterfactual and control groups in the evaluation studies of ATP has provided a growing body of rigorous evidence that ATP is a potent causal factor in the observed impacts. Repeatedly, study findings have shown a leveraging rather than substitutive effect of ATP funding on private sector support of R&D.
295 Competitiveness effects are reported in the last section of this chapter rather than here, because the thrust of the legislated goal of the program is towards the overall competitiveness of U.S. firms relative to foreign firms and not the competitiveness of individual firms.
296 Branscomb et al., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000, p. 2.
297 Ibid., p.6.
298 David Morgenthaler, “Assessing Technical Risk,” in L. M. Branscomb, Kenneth P. Morse, and Michael J. Roberts, eds., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000, p. 106.
299 L. M. Branscomb and Philip E. Auerswald, Between Invention and Innovation: An Analysis of the Funding for Early-Stage Technology Development, NIST GCR 02–841 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).
300 Gompers and Lerner, Capital Formation and Investment in Venture Markets: Implications for the Advanced Technology Program, 1999, p. 20.
301 Ibid., p. 39.
302 Ibid., p. iv.
303 Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 11.
304 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001.
305 Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999, p. 4.
306 David Ayares, “The Five Little Pigs: A Tale of Xeno-Organ Transplant. In Charles W. Wessner, ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000).
307 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, p. 29.
308 Jeanne W. Powell and Karen Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 30.
309 Ibid., p. vi.
310 Ibid., p. 31.
311 Ibid., p. 11.
312 Feldman and Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, pp. 18–19.
313 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 6.
314 Ibid., pp. 10–11.
315 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 41–43.
316 Ibid., p. 42.
317 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 31.
318 Ibid., p. 31.
319 Laidlaw, Acceleration of Technology Development by the Advanced Technology Program: The Experience of 28 Projects Funded in 1991, 1997, p. 28.
320 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001, pp. 36–41.
321 Maryann Feldman and Maryellen Kelley, “Leveraging Research and Development: The Impact of the Advanced Technology Program,” in Charles Wessner ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000).
322 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 7.
323 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 37–40.
324 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, pp. 11–12.
325 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 29.
326 Laidlaw, Acceleration of Technology Development by the Advanced Technology Program: The Experience of 28 Projects Funded in 1991, 1997.
327 Ibid., pp. 15–16.
328 Ibid., pp. 22–23.
329 Ibid., p. 25.
330 Ibid., pp. 28–29.
331 Ibid., pp. 28–29.
332 Ibid., p. 25, pp. 34–35.
333 Ibid., pp. 25–37.
334 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, pp. 1–23.
335 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. iv.
336 The analyst was unable to obtain responses for six of the 50 projects.
337 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 24.
338 Already discussed in Chapter 4 and Chapter 7 was the work of Griliches, Regev, and Trajtenberg with Israeli program data, which showed a positive effect of technology support programs on firm productivity.
339 Darby et al., Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes, 2002, p. 10.
340 Ibid., p. 13.
341 Ibid., p. 19.
342 Ibid., p. 1.
343 Ibid., p. 19.
344 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002.
345 Ibid., p. vi.
346 Ibid., p.vi
347 Ibid., p. vi.
348 Powell, Business Planning and Progress of Small Firms Engaged in Technology Development through the Advanced Technology Program, 1999, with the percentage participation figure updated by Powell in 2002.
349 Ibid., p. 45.
350 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p.17.
351 Knowledge spillovers to others and their subsequent commercial activities provide an indirect path to project impact. For examples of ATP’s direct and indirect paths to impact, see Ruegg, “Delivering Public Benefits with Private-Sector Efficiency through the Advanced Technology Program,” 2000, pp. 118–120.
352 Ibid., Appendix A, pp. 253–258.
353 Ibid., p. 18.
354 Ibid., p. 18.
355 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998, pp. 1–23–1–24.
356 Pelsoci, Closed-Cycle Air Refrigeration Technology for Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, p. ix.
357 Ehlen, Economic Impacts of Flow-Control Machining Technologies: Early Applications in the Automobile Industry, 1999.
358 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 19. comparison. Rather, it merely reported the frequency of collaboration associated with ATP projects, and captured some participants’ comments concerning the benefits.
359 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996.
360 Ibid., p. 25.
361 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 19.
362 Ibid., p. 28.
363 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 19–20.
364 R. Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” in Charles F. Wessner, ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000) pp. 261–262.
365 Ibid, pp. 261–262.
366 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, pp. 12–13.
367 Ibid., p. 5.
368 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001, p. 4.
369 Hall et al., Universities as Research Partners, 2002.
370 Kogut and Gittelman, Public-Private Partnering and Innovation Performance Among U.S. Biotechnology Firms, 2000.
371 Liebeskind, Study of the Management of Intellectual Property in ATP-Grantee Firms, 2000.
372 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997. ATP allows time for projects to find replacement partners when they lose critical project participants.
373 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 33.
374 Responding to project changes requires balancing the need for flexibility to allow firms to make changes needed for project viability, with the need to adhere to ATP’s legislated mandate to fund high-risk research to develop technologies with potential for generating broad-based benefits. To protect the public trust, ATP decides on a case-by-case basis, after reviewing changes in project makeup, whether to approve or disapprove the changes.
375 Because of the relatively small sample size, their findings should be regarded as suggestive and tentative.
376 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002.
377 Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, 1999, pp. 9–17.
378 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, pp. 19–20.
379 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 24.
380 Ibid., pp. 27–30.
381 There is not an exact correspondence between all categories of benefits probed by the two surveys. The BRS used the ratings: “significantly, moderately, little/none, not sure.” Silber used the ratings: “great extent, moderate extent, small extent, no extent.” The comparison shown in Table 9–16 assumes that “significantly” and “great extent” are roughly equivalent.
382 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 20.
383 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 31–32; the list of problems is summarized from Silber’s discussion.
384 Ibid., p. 34
385 Ibid., p. 36.
386 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000, p. 22.
387 In response to early evidence of joint venture.
388 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, p. 31.
389 Ibid., p. 33.
390 Ibid., p. 33.
391 Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996.
392 Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, 1996.
393 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000; and Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2, 2001.
394 Feldman and Kelley, Winning an Award from the Advanced Technology program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2000, p. 11.
395 Ibid., p. 14.
396 Fogarty et al., ATP and the U.S. Innovation System—A Methodology for Identifying Enabling R&D Spillover Networks with Applications to Micro-electromechanical Systems (MEMS) and Optical Recording, 2000.
397 Cohen and Walsh, R&D Spillovers, Appropriability and R&D Intensity: A Survey Based Approach, 2000.
398 Economic rent refers to the excess of return over a competitive rate of return.
399 Martin et al., A Framework for Estimating the National Economic Benefits of ATP Funding of Medical Technologies, 1998.
400 Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002.
401 Austin and Macauley, Estimating Future Consumer Benefits from ATP-Funded Innovation: The Case of Digital Data Storage, 2000.
402 Chang, A New Lexicon and Framework for Analyzing the Internal Structures of the U.S. Advanced Technology Program and its Analogues Around the World, 1998.
403 Ibid., p. 67.
404 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity ofParticipating Firms: A Framework Using Both U.S. and Japanese Data, 2002.
405 Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, 2000.
406 Schachtel and Feldman, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. 1, 2000, p. 1.
407 Feldman et al., Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. 2, 2000.
408 Solomon Associates, The Advanced Technology Program, An Assessment of Short- Term Impacts: First Competition Participants, 1993, p. 21.
409 Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, pp. 46–47.
410 Powell and Lellock, Development, Commercialization, and Diffusion of Enabling Technologies: Progress Report, 2000.
412 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997.
413 Ibid., p. 6.
414 Ibid., pp. 30–31.
415 Ibid., p. 30.
416 Darby et al., Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes, 2002.
417 That a terminated project may yield knowledge gains even if it fails to achieve its ultimate goals is demonstrated by a case study of a terminated project provided in Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, pp. 261–265.
418 Ibid., p. 260.
419 Ibid., pp. 259–260.
420 Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” 2000, pp. 259–277.
421 Advanced Technology Program, Performance of 50 Completed ATP Projects Status Report 2, 2001, p. 29–21.
422 Ibid., p. 25.
423 Ibid., p. 21–22.424 Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997.
425 Pelsoci, Closed-Cycle Air Refrigeration Technology For Cross Cutting Applications in Food Processing, Volatile Organic Compound Recovery, and Liquid Natural Gas Industries, Economic Case Study of an ATP-Funded Project, 2002, pp. vii-ix.
426 Advanced Technology Program, Performance of 50 Completed ATP Projects, Status Report 2 , 2001, pp. 22–23.
427 Wessner, The Advanced Technology Program: Assessing Outcomes, 2000.
428 Ibid., p. 94.
429 Chapter 9 in its entirety is a summary of the findings of many studies. Hence, this end-of-chapter summary provides a higher level overview of the results of the chapter.
Date created: July
NIST is an agency of the U.S. Commerce Department