A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 4:  Modeling and Informing Underlying Program Theory

As discussed in Chapter 2, the modeling and informing underlying program theory is an essential prelude to program operations, an early stage complement to other management and evaluation efforts, a building block in the design of a long-term comprehensive evaluation program, and an ongoing source of continuous organizational learning. Clarifying and validating a program’s underlying concepts and the analytical linkages among its various elements is an important part of an agency’s overall program evaluation strategy. Yet, pressures for quick responses to questions about program activities can lead program administrators to neglect program theory and the investigation of program dynamics.

At ATP, modeling and informing underlying program theory has yielded information critical to the program’s survival, shape, and success, and has become a mainstay of ATP’s evaluative program. Identifying the characteristics of multiple, complex, causal paths has been an important part of ATP’s evaluation plan because program modeling strategy involves influencing both the immediate and proximate determinants of firm and industry behavior. The combination of external challenges and internal commitment to documentation of results has produced a program noteworthy among federal government programs in the level of attention it has devoted to understanding and documenting the program’s effects.

This chapter details ATP’s use of analytical and conceptual modeling to explore basic concepts and of underlying program relationships, to condition expectations and set benchmarks for performance, and to refine dominant theoretical paradigms. Unavoidably, the numerous relevant concepts and subtopics drawn from multiple sources add complexity to the treatment. To assist the reader, the major themes and the reports and papers used to amplify each theme are listed in Table 4–1, presented in the order discussed in the chapter, and with the particular contribution to underlying concepts and theory noted. The primary purpose of several of the documents cited here was not to explicate underlying program theory, yet they are included because they also made significant contributions to understanding ATP’s workings. Table 4–1 is intended as a quick reference and roadmap through the chapter.

Table 4–1. Twenty-Two Studies and Papers Modeling and

*Note: Three of the studies are listed more than once because they address multiple concepts.

Concepts, Models, Metrics, and Paths Connecting Program Activities to Intended Impacts

Recognizing that it fell between the conventional poles of federal support of basic research on the one hand, and of mission-oriented basic and applied research on the other, ATP quickly sought to clarify its basic concepts, to help explain how it works, and to develop improved models for analysis. A variety of seminars, roundtables, staff papers, presentations, and commissioned studies were used to flesh out concepts, models, and pathways of the program, and point toward the best evaluation methods.

Economic Spillovers

The concept of spillovers is central to ATP and implicit in the argument that ATP will help address under-investment in generic technologies by the marketplace. An “economic spillover” ⁶⁰ is the excess return to society of an investment over the private return captured by the investing firms. The presence of particularly large spillovers may cause private firms to invest less than is socially optimal because too much of the benefit escapes them. Spillovers provide a primary justification for public intervention to increase investment in R&D. A closely related concept is “inappropriability,” which refers to the inability of an investor to capture, or “appropriate,” the fruits of his or her investment.

To delve deeper into the subject of spillovers and their implications for the program, ATP commissioned Adam Jaffe, economics professor at Brandeis University and member of the National Bureau of Economic Research (NBER), to prepare an economic analysis of research spillovers. ⁶¹ Noting that spillovers have been of interest to economists for several hundred years, Jaffe pointed out that R&D activities of private firms have been shown to generate spillover benefits; hence, ATP should be expected not only to fund R&D that generates spillover benefits, but R&D that yields higher-than-average spillovers.

Jaffe’s work identified three different sources of spillovers relevant to ATP— knowledge spillovers, market spillovers, and network spillovers—and noted that the three interact synergistically to increase their combined effect. Table 4–2 defines each source of spillovers and gives examples of how each occurs. Figure 4–1 illustrates how knowledge and market spillovers interact and may lead to a gap between social and private returns.

Table 4–2. Classification of Spillovers

Source: Summarized from Jaffe, Economic Analysis of Research Spillovers, 1996.

Figure 4–1. Private and Social Returns to R&D: Pure Market Spillover, Plus Pure Knowledge Spillover, Plus Interaction of the Two

Firm 1 invests in R&D, generating new knowledge that it uses to improve its products or lower its production costs. Assuming the firm successfully commercializes the results, market competition causes the value of some of firm 1’s improvements to be captured by its customers in the form of lower prices or higher quality. This effect alone would cause a spillover gap equal to the customer benefit. But the figure shows other effects. The first downward pointing arrow indicates that knowledge spillovers flow from firm 1’s knowledge base to other firms through disembodied outputs such as papers and patents. The second downward pointing arrow indicates that knowledge also passes from firm 1 to other firms through research results embodied in its new commercial products and processes. The third arrow, which points upward, indicates that at least some of the firms benefiting from the knowledge spillovers are competitors of firm 1, who then introduce cheaper or better products into firm 1’s markets, taking some of its profits and creating some additional customer benefits. Meanwhile, these other firms may also introduce improved or lower cost products and process into their own markets, resulting in firm profits and customer benefits. As Jaffe observes, “...the combination of knowledge spillover with competitive interaction increases the spillover gap both by raising the social return and lowering the private return.” (p. 17)

Source: Jaffe, Economic Analysis of Research Spillovers, 1996.

In drawing the implications of spillovers for ATP, Jaffe identified factors that he saw associated with higher spillovers—a desirable feature from ATP’s point of view. Table 4–3 lists some of these factors. According to Jaffe, ATP could improve its ability to select projects with greater-than-average spillover potential by acting on this information. This study gave ATP a model and specific guidance for increasing its effectiveness.

Table 4–3. Factors Increasing the Likelihood of Spillovers

Source: Compiled from information presented in Jaffe, Economic Analysis of Research Spillovers, 1996, pp. 42–44.

Descriptive Models of ATP

Models have more than analytical value; they can also help communicate a program to stakeholders. ATP staff developed several models to help explain the program’s mission to a diverse audience. These models were used to explain and discuss the program in settings such as policy forums, internal strategic planning sessions, and ATP workshops.

The Three Dimensions of ATP’s Mission

Modeling by J-C Spender, UK Open University Business School, helped ATP solidify its internal perspective and explain its dynamics. ⁶² The three dimensions shown in Figure 4–2 embody three main ideas that underlay ATP’s enabling legislation: scientific and technical knowledge gains (axis labeled Technical Knowledge), commercial gains and improved competitiveness of U.S. companies (axis labeled Private Returns), and broadly based spillover benefits to the nation at large (axis labeled Public Gains). The figure is a conceptual illustration, not intended for quantification.

In describing the program, Spender, the diagram’s author, speaks of ATP’s goal as “promoting trajectories through an innovation space formed by partnership between the USA’s three institutionally distinct modes of scientific and technological innovation: scientific research; private enterprise; and, public-sector management of society’s public goods.” ⁶³

Spender’s framework also has applicability in addressing critics’ complaint that ATP represented “corporate welfare.” According to Spender, charging that ATP is “corporate welfare” is to mistake, with reference to Figure 4–2, the point P for the point Q:

Critics are correct in the sense that market-driven firms are being subsidized and are therefore benefiting, but off the mark in the sense that the point P is incidental. It indicates only the agent’s incentive to act on ATP’s behalf, not the social result of its activity. Ultimately the project’s success must be measured in terms of the point Q (p. 49).

Figure 4–2. Three Dimensions of ATP and the “Journey to Q”

In concept, technical progress is measured in the diagram along the axis labeled Technical Knowledge, net benefits to the awardee from commercial activity along the axis labeled Private Returns, and net spillover benefits to others along the axis labeled Public Gain. The dimensions’ orthogonality illustrates the independence of these outcomes (i.e., technological advance does not necessarily lead to economic activity, and commercial activity does not necessarily lead to public interests). The point Q depicts a desirable outcome for ATP—the end point of a project trajectory producing a combination of knowledge gains and net benefits to the awardee that are exceeded by large net spillover benefits to others in the economy.

Source: Spender, “Publicly Supported Non-Defense R&D: The U.S.A.’s Advanced Technology Program,” 1997.

In addition to being useful in explaining the program’s design to external constituencies, Spender’s model also helped to develop a common perspective among ATP’s functional staff: primarily scientists and technologists, business specialists, and economists. In the early to mid-1990s, combining staff from these fields in a single unit was still relatively new in business and government, and there was a distinct learning curve for the staff. As Spender puts it:

ATP draws the participants into a common framework even if the innovative activity may not be comprehensible to those not institutionalized into the appropriate professional environment. Business experts are unlikely to appreciate the niceties of the research scientist’s problems or respect the professional judgments required, especially where the levels of technical risk are high. Likewise, even though ATP seeks low-risk commercialization and public-goods effects, the complexities and subtleties of the social and economic works in which these arise are often puzzling to scientists. (p. 50)

ATP’s Two Paths to Impact

Looking for a more succinct way to explain the program to diverse audiences, and specifically to provide context for examples of project impacts, ATP began to use the “two-path” diagram reproduced in Figure 4–3, which depicts direct and indirect paths to program impact. ⁶⁴ The “direct path” represents the commercialization route of the award-recipient and its close collaborators to capture private benefits and generate market spillovers, as well as possible knowledge spillovers as others reverse engineer their products and processes. The “indirect path” represents the main route to knowledge spillovers: award-recipient publications, presentations, interactions, patents, and mobility of researchers among organizations, allowing others to gain knowledge from the funded research without paying for it. If these other organizations use the knowledge obtained for economic gain, benefits from knowledge spillovers result. ⁶⁵

The direct path has special significance because it allows ATP directly to encourage U.S. businesses to accelerate development, commercialization, and use of new technologies. The indirect path may ultimately prove to be even more important than the direct path in its generation of benefits, but it tends to ⁶⁶ Nevertheless, ATP can foster the generation of knowledge spillovers by selecting generic technologies applicable to many firms both upstream and downstream and by supporting publishing, patenting, collaborative activities, and other activities through which knowledge is diffused.

The diagram highlights the extension of ATP’s role beyond that of technical knowledge generator/disseminator. By engaging the activities of for-profit U.S. firms and requiring them to plan up-front for commercialization and establishment of a pathway to early application of the technology, ATP seeks to meet its mandate to generate accelerated economic benefits and improved competitiveness of U.S. firms in international markets.

Planning Performance Metrics for ATP

An important component of ATP’s evaluation program has been to convert the concepts that emerged from modeling its underlying theory into variables amenable to measurement/quantification. In an early effort along these lines, ATP supported the work of Albert Link of the University of North Carolina, Greensboro, to recommend performance metrics for ATP. ⁶⁷ Link also provided advice on developing an implementation strategy for compiling the metrics.

Table 4–4 summarizes Link’s recommended set of measures and timetable for collecting them. The nuts-and-bolts data collection plan proved useful for ATP as it sought to implement its evaluation activities quickly. In a very practical way, Link’s plan informed the theory and pathways of program effect.

Figure 4–3. Direct and Indirect Paths to ATP’s Impacts

If a project’s participating companies successfully take the technology into commercialization (direct path), this is expected to speed the introduction of the technology by U.S. companies. If participants falter in the commercial phase, then they may nevertheless leave behind publications, patents, informed workers, and other means of transferring the knowledge to others who may take it forward. If activities along both paths are successful, then the gains can be even greater, because the social return of the project is the net impact from the combination of the two paths and their interactions.

Source: Ruegg, “Delivering Public Benefits with Private-Sector Efficiency through the Advanced Technology Program,” 2000.

Table 4–4. Recommended Metrics for ATP and Timing of Data Collection

Source: Link, “Measuring the Economic Impact of the Advanced Technology Program: A Planning Study,” 1992, p. 30.

ATP routinely collected much of the data listed in Table 4–4. In the mid-1990s, ATP expanded and revised its data collection efforts to be more responsive to Government Performance and Results Act of 1993 (GPRA) reporting requirements and stakeholder questions. ⁶⁸

Understanding Relationships Between Program Design Features and Outcomes

As noted, relationships between program design features and program outcomes are often sketchy. The discovery of systematic differences in the effectiveness of design-feature variations could improve funding decisions. For example, ATP’s enabling legislation emphasizes the role of research joint ventures in achieving ATP’s mission, but does not suggest which types of partnering arrangements may be most conducive to high levels of program performance. A better understanding of what makes joint ventures work could be useful in guiding ATP’s efforts. To this end, ATP supported a study by Jeffrey Dyer, Brigham Young University, and Benjamin Powell, University of Pennsylvania, to investigate factors that increase or decrease the likelihood of success of R&D collaborations. ⁶⁹, ⁷⁰

Dyer and Powell conducted semi-structured interviews with companies participating in 18 ATP-funded joint ventures developing technologies with application in the automotive industry. Their interview discussions centered on the questions listed in Table 4–5.

They found that greater knowledge sharing and more effective coordination among participants characterized the more successful joint ventures. They identified several factors that influence the extent to which participants share knowledge, and that influence the costs of coordinating the venture’s activities. ⁷¹ Their diagram showing the key factors that participants said influenced the degree of success of their joint ventures, the direction and nature of impact, and the linkages from these factors to measures of outcome success is shown in Chapter 9’s report on findings, Figure 9–4. Understanding better how the program contributes to collaborative success and which factors can make or break a joint venture are important for ATP administrators and program managers. For instance, Dyer and Powell documented the importance of face-to-face meetings of participants to build trust, and the critical role of trust in collaborative success. As standard procedure in the contracting process, ATP staff must approve proposed travel budgets for which ATP funds are to be used. So, without an understanding of this underlying success factor, well-meaning staff members could easily make what they intend as “efficient” budgetary decisions (e.g., refusing to approve travel costs for frequent face-to-face meetings among joint venture participants), which may have a negative effect on the project’s success.

Table 4–5. Interview Questions Investigating Determinants of Success of Joint Ventures

Source: Dyer and Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, 2002.

Conditioning Expectations through Studies of Private-Sector Behavior

Modeling private-sector decisions and outcomes has helped define expectations of what the program can achieve and helped establish benchmarks for ATP’s performance, especially in shaping realistic expectations of the technical and economic successes of a firm’s R&D portfolio. Also, of use for ATP has been knowledge about private-sector tools for managing risk, as well as analysis of private-sector decisions in the face of risk, and evolving relationships among firms, universities, and other organizations. A better understanding of barriers to private investment in early stage, science-based innovations sheds light on ATP’s value-added role in financing of R&D projects.

Social Rates of Return from Business R&D

Early in its evaluation program, ATP explored the applicability of a widely used model developed in the 1970’s by Edwin Mansfield, University of Pennsylvania, to measure market spillovers of R&D. ⁷² ATP also explored the implications of Mansfield’s earlier application of his model to estimate private and social returns from new products or processes when private-sector innovators commercialize their technologies. ⁷³ As Mansfield writes:

From the point of view of the ATP program, the gap between social and private rates of return from investments in new technology is of central importance. After all, a major rationale for the ATP program is that some R&D projects have social rates of return far in excess of private rates of return. (pp. 26–27)

Mansfield’s Model

Mansfield measured resulting market spillovers in terms of “consumer surplus.” In Mansfield’s approach, the social benefits from an innovation are measured by the profits of the innovator from the innovation plus the benefits to consumers following reduction in the price of the good due to the innovation. ⁷⁴ Figure 4–4 illustrates his basic model.

Figure 4–4. Social Benefits from Product Innovation that Reduces Costs of Industries Using It

Assume that the supply curve was S1 before the innovation, and the price charged by the industry was P1. After the introduction of the innovation, the supply curve is S2, and the price charged is P2. The social benefits from the innovation are measured as the sum of the two shaded areas shown in Figure 4–4. The top shaded area is the consumer surplus due to the lower price, P2 rather than P1. The bottom-shaded area in Figure 4–4 is an estimate of the additional resource savings from the innovation. A resource savings results, leading to a corresponding increase in output elsewhere in the economy, because the resource costs of producing the good after the innovation are P2 Q2, minus the profits (r) the innovator receives from the innovation. The amount r is merely a transfer from the producers of the good using the innovation to the innovator. On net, there is consumer surplus from the price reduction and a resource savings amounting to the profits of the innovator. (Caveats are given by Mansfield in his report.)

Source: Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996, p. 20.

Mansfield’s findings on the characteristics of industrial R&D decision making are drawn from interviews with industrial R&D officials. The interviews point to the difficulty that even the most sophisticated firms have in forecasting the private returns from company-financed R&D projects. Despite the use of and search for formal decision-making models such as scoring systems, programming techniques, and other quantitative decision analysis techniques, “available studies,” according to Mansfield, “indicate that few firms, if any, are confident of their forecasts of particular R&D projects.” ⁷⁵ The reasons for this limited ability to accurately forecast rates of return relate to the inherent uncertainties of R&D projects, and to the difficulty that firms have had forecasting development cost and time, the probability of success, and the profitability of new products or processes. According to Mansfield’s analysis, firms face an additional difficulty in “forecasting how R&D, if successful, will be utilized.” ⁷⁶ Figure 4–5 illustrates that there is difficulty in trying to forecast private returns from investments in new technology early in the development process, and that the accuracy improves over time. The figure shows the extent of the forecasting errors within a single company for 57 new processes or products as a function of the number of elapsed years after their development. Shortly after the new processes or products were developed, the profit forecasting errors were high. As time passed, estimates of discounted profits were revised each year to the point that they became very close to actual discounted profits. The errors were eliminated through revision over time, but the error rate in the initial estimates was not found to improve over time.

Application of Mansfield’s Model to ATP

Several factors limit application of Mansfield’s model to ATP. Mansfield applied his model to single products, whereas ATP wanted to apply it to technology platforms generating multiple products. He used his model to capture only market spillovers, whereas ATP wanted to capture knowledge and network spillovers in addition to market spillovers. Finally, in estimating the model, Mansfield applied his model using many years of historical data, whereas ATP technologies were in the early stages of commercialization.

Figure 4–5. Reducing Inaccuracies in Profitability Estimates Over Time

^a Computed as the proportion of product/process cases for which the ratio of the profit forecasts made in each successive year after development to actual discounted profits is equal to or greater than 2.0 or less than or equal to 0.5.

Source: Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996, p. 16.

To overcome these limitations, Mansfield recommended applying his model multiple times to capture market spillovers of enabling technologies with multiple applications. To meet concerns about the large forecasting errors early in the life of a new product and the fact that these errors tend to diminish rapidly as time passes, Mansfield recommended repeating and updating a set of case studies several times to see how the results changed over time. The commissioned a set of cases with Mansfield to be done according to his recommendations, recognizing that supplemental techniques would be needed to capture knowledge spillovers omitted from Mansfield’s model. ⁷⁷

Other case studies of ATP projects have followed Mansfield’s basic approach. See, for example, the calculation of social and private rates of return on medical technologies performed by Research Triangle Institute, discussed in Chapter 6.

Implications of Mansfield’s Model for ATP

Mansfield’s model provided a starting point for modeling the social and private rates of return of ATP-funded projects, and the empirical results from his work provided an initial benchmark for ATP’s assessment of the effects of its awards. Mansfield’s work, conducted in the late 1970s, included estimates of the private and social rates of return of 17 private-sector innovations. In this study he collected data on project costs, revenues, and profits. He then collected similar data from other firms in the same industry for similar products and processes they introduced, data from firms purchasing or licensing the new products or processes, and data from final users of consumer goods. From these data he calculated the private and social rates of return on the innovations. Of particular interest to ATP, Mansfield found significant market spillover benefits to consumers, even when the private returns were low or negative. The results became well known among economists, and have conditioned general expectation about the relationship between social and private rates of returns from technological innovations.

The results, in effect, “up the ante” for ATP; funding projects that yield positive market spillovers would not be sufficient for program success because “routine” private-sector innovation could do that. The existence of positive market spillovers from private-sector innovation underlies Jaffe’s observation, as noted above, that ATP would have to fund projects that yield higher-thannormal spillover effects if it were to add value as a public-sector program. This observation had important programmatic implications for ATP, because it led to a search for innovative projects with the potential to create substantial social benefits while meeting private-sector criteria necessary to gain participation by private firms.

Adequacy of Private Investment

The 1980s and 1990s saw a substantial increase in venture capital funds and investment funds available from “angel” investors. The larger amounts of venture funds were largely the result of a change in law that no longer prohibited institutional investors from venture investing. ⁷⁸ The larger amounts of angel funding were attributable in part to the dramatic increase in personal wealth in the United States over the same period. Table 4–6 traces the amount of venture capital raised each year from 1977 through 1995, as well as the amount of venture funding going to early stage investments. Nearly three times as much went to early stage investments in 1995 as in 1977, though at the time of preparing this report there has been a cyclical contraction in venture funding.

The increase in private sector supply of venture capital funding was one of the facts cited to challenge the need for government funding of high-risk technological innovations, and thus ATP. To explore this issue, ATP commissioned an inquiry into the supply of venture capital to fund enabling, high-risk technologies.

Paul Gompers and Josh Lerner, both professors at Harvard University Business School and members of the NBER, conducted the study, investigating the trends and patterns of venture capital and angel funding available to small innovative firms. ⁷⁹ Gompers and Lerner also conducted seven case studies of small R&Dintensive firms funded by ATP to determine why those firms needed ATP funding and the role the funding played. Several of their findings help to explain why, despite the large increase in private venture funds, projects of the type targeted by ATP can go wanting. ⁸⁰

Putting the supply of venture capital in perspective, the researchers found that despite the increases in venture capital supply, less than one tenth of 1% of business startups annually have received venture financing in recent years. They described lemming-like behavior on the part of many venture capitalists, leading to a concentration of investments in “hot” technical areas, while other areas attract little or no venture capital. They identified geographical concentration of the bulk of the venture capital, noting that companies in many parts of the nation receive little or no venture capital. They suggested that the shift in the source of venture capital away from the individual investor and toward the institutional investor means that there is a greater preference for less risky R&D and shorter time horizons for realizing returns on these investments. They suggested that investors, for management purposes, have tended to increase the size of individual investments, rather than increase the number of investments in proportion to the growth of funds available. Hence, a doubling in the amount of venture funds available does not result in a doubling in the number of projects funded, but rather an increase in the size of the average project, as a fund manager can better stretch limited managerial resources over a portfolio with a higher dollar value than one that contains more projects. From their case studies, described further in Chapter 6, the authors concluded that each of the companies examined struggled to obtain funding to undertake its innovative research and was unable to secure sufficient funding from private sources.

Table 4–6. Volume of Venture Capital Activity

N/A = Data not available.

Source: Gompers and Lerner, Capital Formation and Investment in Venture Markets: Implications for the Advanced Technology Program, 1999, p. 17.

Private Firm R&D Decisions in the Face of Risk

To define the role technical risk plays in the financing decisions of firms and to explore further the adequacy of funding for developing high-risk, enabling technologies, ATP commissioned Harvard’s John F. Kennedy School of Government, in collaboration with MIT’s Sloan School of Management and the Harvard Business School, to conduct a broad ranging study. ⁸¹The Harvard-MIT Project on Managing Technical Risk examined barriers to private investment in earlystage, high-risk technology development projects. The study’s objective was to assist ATP to “better identify projects that would not be pursued or would be pursued less vigorously without ATP support and at the same time are likely to lead to commercial success—with broad public benefits—with that support.” ⁸²

The Harvard-MIT study drew primarily on expert assessments offered by industrial, academic, financial, and government representatives participating in a series of workshops held between 1999 and 2001. The workshops were organized around discussions based on a set of papers commissioned by academic participants and practitioners.

The study, in part, updated Mansfield’s summary of the techniques, behaviors, and perceptions of R&D managers toward technical and economic risk. The Harvard-MIT project examined the institutional, behavioral, financial, and nonfinancial barriers that produced inadequate incentives for entrepreneurs, venture capitalists, and corporations “to undertake some varieties of early-stage, high-risk technology development projects that have potential to generate radically new products and processes.” ⁸³

Managing Risk

Several themes about risk and its effect on financing technology emerged from the Harvard-MIT workshops. Industry R&D participants emphasized the art of quantifying technical risk. Further, although participants noted the existence of numerous well-established methodologies for assessing technical risk, their consensus was that none of these methods was very successful. ⁸⁴ Still, they agreed that understanding the sources of risk and dealing with them systematically is important.

Participants expressed varying perspectives on the ability to separate technical risk from market risk. As posed by the project’s report, “In a radical technical innovation, can one expect to define product and process specifications, then engage in research that is sufficient to reduce technical uncertainties to an acceptable level?” ⁸⁵

The problem here, as described by several industrial representatives, is that the desired market “specifications” about the characteristics of the intended product can be unstable. Consumer requirements may change during the project. Independently, the performance of the technology may evolve differently than predicted. “Those differences require an adjustment in the specifications, which in turn requires that market estimates be adjusted, which in turn may suggest a further adjustment in product specifications.” ⁸⁶

Institutional Factors

As part of the Harvard-MIT study, a commissioned paper by Scott Shane, University of Maryland, tested the proposition that newly created firms are “a particularly appropriate institutional form within which to make success of radical, science-based innovations,” ⁸⁷ while a commissioned paper by James McGroddy found that “a large firm with deep technical roots has some advantages over the startup with limited resources, in that the former can incubate the technology for several years before bringing it to market, thus reducing substantially the uncertainties surrounding the technical challenges.” ⁸⁸ Generally, the Harvard-MIT study highlighted an increased inter-firm and inter-sector dependence of purchasers and suppliers of both R&D and goods on one another.

The findings and assessments presented in the Harvard-MIT study suggest that the process of commercializing promising technical concepts involves more than public sector funding of pre-commercial R&D. Rather, “institutional” factors as much as “economic” factors may hinder private investment in early stage, sciencebased innovations. This finding poses new questions and program design issues for ATP. Thus, according to the study:

...major changes are transforming the institutional structure of the high-tech industrial economy. Large corporations are increasingly focusing on their role as system integrators, low-cost producers, and distributors and marketers internationally, while outsourcing much of their innovation to mid-size and smaller, technically specialized firms in their supply chain. Where will those small-to-medium size firms get their insights into the art of the possible from new science, if not from the large firms they serve? Is this another reason for public programs like ATP? (p. 6)

“Valley of Death”

The Harvard-MIT study also addressed the adequacy of private-sector funding of high-risk R&D projects. Whereas the Gompers-Lerner study addressed the overall availability of venture capital, the Harvard-MIT study cited a striking degree of general agreement among technical entrepreneurs, high-tech business managers, and venture capital investors about the existence of a “Valley of Death”—a significant gap between federally funded basic research and industry-funded applied research and development.

According to the Harvard-MIT study: ⁸⁹

Congress appears to have intended the ATP to be a mechanism for addressing the “Valley of Death” area of research, which has been described by Charles Vest, president of MIT, as “mid-level” research, and by Branscomb as “basic technology research.”

If such a gap is real, the fact that growth of commercial R&D is outstripping the growth of government-funded [R&D] only means that the gap may be growing, not shrinking, as public funding of early stage research fails to keep pace with commercial development. (p. 49)

Positioning ATP in Capital Markets

There are a number of federal programs that provide grants, loans, and other forms of assistance to U.S. firms. One of the challenges of a program like ATP is to convey how it differs from other programs and the circumstances under which it is a suitable or unsuitable funding source for a given organization. This communication necessarily takes place with busy CEOs, CFOs, and company researchers against a cacophony of background noise. Clarity is critical.

Figure 4–6 reproduces a framework that locates ATP in the broader financing landscape. ⁹⁰ It is part of a planning guide prepared for use by ATP award recipients in the post-award period, but it also serves as a broader outreach tool for ATP.

The guide, an important tool of communication with ATP’s direct stakeholders, models complex business strategies for the technology platforms envisioned as downstream results of the R&D projects. Figure 4–7 illustrates ATP’s goal to foster creation of technology platforms that can spawn multiple business opportunities. The guide also covers teaming arrangements, and many other concepts critical to the awardees achieving not only business success, but also project success.

Figure 4–6. Roadmap of Financing Options

The figure shows five categories of financing in a financing roadmap. As indicated by the arrow and side note, ATP awardees are seeking an R&D partnership. Federal, state, and private sources provide financing opportunities through partnerships. The figure distinguishes ATP’s role from the other partnership opportunities. The figure also points to alternative sources of financing that award recipients can attempt to access as the ATP funding is drawn down. The Commercialization and Business Planning Guide positions ATP in the financing landscape and assists startup companies who need help with the ins and outs of seeking and obtaining financing from different sources.

Source: Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 20.

Conditioning Expectations through Studies of Other Public Sector Programs

With new programs come new expectations, and defining reasonable expectations is critical to a new program's success. Exmining state programs and counterpart programs abroad has helped condition general expectations about ATP.

Figure 4–7. Strategic Mind Map for a Rich Technology Platform

The figure shows an example of a company that has developed a portfolio of intellectual property to protect its process for creating a new material as well as the devices used to produce the material. The figure illustrates partitioning of multifaceted opportunities, taking into account downstream and upstream potential applications, intellectual property positioning, and economic factors. It shows multiple licensing opportunities (i.e., to license a process to produce material, to license production equipment, and to license in application-specific product market) and multiple application areas.

Source: Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 53.

Interface of ATP with State Programs

In the United States, the same macroeconomic and international economic trends that gave rise to the establishment of ATP also led to a rethinking by state governments of how best to promote state economic development. Many states began taking more active steps to revitalize their economies. ⁹² This led to a modest shift in emphasis from traditional state recruitment strategies that centered on bidding for existing firms through monetary incentives, such as reduced taxes, to one on generating new firms and industries. ⁹¹

Beginning in the early 1980s, a growing number of states established state technology programs. Varying somewhat in emphasis across the basic research, applied research, and commercialization continuum, these programs included a diverse set of specific programs: incubators, venture capital funds, manufacturing modernization programs, cooperative university-industry R&D programs catalyzed and subsidized by state funds, and others. ⁹³

The parallel development of state and federal government technology programs throughout the 1980s created new opportunities as well as complexities in coordination. Several of the major federal initiatives, such as the National Institute of Standards and Technology’s Manufacturing Extension Partnership program and the National Science Foundation’s Engineering Research Centers program, required state contributions, but ATP did not. Consequently, few direct linkages existed at the outset between ATP and the state programs.

To examine the relevance of state technology development programs to its operations, ATP commissioned a two-volume study of state experiences, under the title, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, Volume 1: A Guide to State Business Assistance Programs for New Technology Creation and Commercialization, is authored by Marsha Schachtel, a Senior Fellow at the Johns Hopkins Institute for Policy Studies, and Maryann Feldman, then Associate Professor of Economics at Johns Hopkins University and Director of its Information Security Institute, and now a professor in the Rotman School of Management at the University of Toronto. Volume 2, Case Studies of Technology Pioneering Startup Companies and Their Use of State and Federal Programs, is authored by Maryann Feldman, Maryellen Kelley, then of ATP and later of Pamet Hill Associates, Joshua Schaff, Director of the New York City Democracy Network, and Gabriel Farkas, graduate student at Dartmouth College. The second volume presented four case studies describing how ATP award winners used state government programs in combination with ATP assistance. It is discussed in greater detail in Chapter 6. ⁹⁴

Volume 1 analyzed the structure of the state programs, and thus provides a benchmark against which both the distinctive and complementary roles of ATP can be examined. It described the range of state services available both to applicants and awardees, such as guided access to technical information, patent search assistance, technical assistance from university and extension agents, and others, along with examples of specific state programs. Of direct relevance to ATP’s evaluation efforts was the report’s delineation of the challenges that underlay efforts by the private sector to commercialize technology, and of the place of state government, and implicitly, federal government efforts in assisting private firms. Volume 1 serves as a guide to award recipients to “the type of state resources available to help them carry out the commercialization plans outlined in their project proposal, grow their businesses, and eventually successfully diffuse the technologies developed with ATP’s financial assistance.” ⁹⁵

The framework provided in Table 4–7 helped define the scope of ATP’s involvement in the multiple aspects of developing and commercializing technological innovations relative to the scope of involvement by the states. This definition of what ATP is and what it is not, is a necessary element in specifying the criteria against which the program is to be evaluated. By identifying the range and content of the panoply of state technology programs available to ATP applicants and awardees, the Schachtel-Feldman study pointed to the presence of complementary variables linking ATP support to specific outcomes. In effect, for some set of awardees the impacts of ATP grants are moderated by whether or not they receive supplemental state support in any of the nine cells described above. This statement leads to a previously unidentified evaluation question: Do ATP awardees who receive support from state technology programs generate different technical or economic outcomes than those firms which did not receive such supplemental assistance? This unanswered question has implications for the ways in which ATP seeks to couple its activities with those of state programs.

Comparisons of ATP with Counterpart Programs Abroad

ATP was established after a number of similar programs in other industrialized countries were up and running. According to three notable economists, Zvi Griliches, Haim Regev, and Manuel Trajtenberg: ⁹⁶

Most OECD [Organization for Economic Cooperation and Development] countries have government-funded R&D support programs that encourage private investment in civilian R&D. There seems to be a rather wide consensus among economists in the OECD countries that the social rate of return to R&D investment substantially exceeds the private rate of return, suggesting that government action to promote research and development enhances economic welfare. (p. 2)

ATP’s interest in counterpart programs in other countries was motivated by two objectives: to learn from these programs in order to make ATP a better program and to implement a statutory requirement pertaining in part to foreign counterpart programs. The requirement is that ATP include in its funding consideration the participation of U.S.-based subsidiaries of foreign-owned companies only if their participation passes three additional tests beyond the selection criteria to which all proposals are subject. One of the tests is that “the parent company is incorporated in a country which affords to United States-owned companies opportunities comparable to those afforded to any other company, to participate in any joint venture similar to those authorized under this chapter….” ⁹⁷ To apply this provision requires that ATP keep abreast of counterpart programs in other nations and to apply the country-specific provisions to determine eligibility whenever a proposal from a foreign-owned subsidiary reaches the semifinalist stage in an ATP competition. ⁹⁸

ATP’s interest in comparative programs gave rise to analytical studies, briefings and exchanges with foreign visitors, and participation in international conferences and forums. Although not described here in detail, conferences provide productive opportunities for ATP staff to review program and operational features with their counterparts in other countries.

Table 4–7. Categorizing State Technology Assistance Program by Type of Challenge and Stage of Technology Development and Commercialization

This framework is derived from Randall Goldsmith’s model of product commercialization. Analytically, each cell in the framework corresponds to a specific possible combination of private sector-public sector (state/federal) relationships. The framework provides Schachtel and Feldman a means of organizing descriptions of the diverse set of state technology development programs in place by the late 1990s.

Their use of the Goldsmith model also has more expansive purposes. It helps describe and implicitly delimit the relative areas of R&D emphasis of ATP and state government programs. State technology programs span the R&D continuum from support of basic research and human capital development via grants to university faculty for research and support of graduate students (e.g., Texas), through support of generic/precompetitive research (e.g., Ohio and New Jersey), to an emphasis on spawning spin-off firms and product development (e.g., Connecticut and Pennsylvania), with some states having some of all of the above. Collectively then, an account of state programs is likely to fill up the Goldsmith cells. The ATP, by way of contrast, has centered its activities on technical challenges, supporting work primarily in the concept and development phases—just two of the nine boxes in the Goldsmith framework.

Source: Schachtel and Feldman, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, 2000, p. 3. Matrix adapted from H. Randall Goldsmith, “A Model for Product Commercialization,” Oklahoma Alliance for Manufacturing Excellence, Tulsa, OK, 1995. ATP’s interest in comparative programs gave rise to analytical studies, briefings and exchanges with foreign visitors, and participation in international conferences and forums. Although not described here in detail, conferences provide productive opportunities for ATP staff to review program and operational features with their counterparts in other countries.

A Common Lexicon and Framework for Making Comparisons

While many external similarities exist between ATP and the technology support programs of other countries, differences also exist. Direct comparisons that do not account for these differences are of limited value in learning about performance. To address this problem and to improve understanding of how these programs operate, ATP economist Connie Chang developed a lexicon for discussing program design features and analyzing their structures. She then applied the lexicon to a sample of ATP-like programs abroad and to a sample of their features. These findings are illustrated in Chapter 9, Table 9–21. ⁹⁹

Chang’s work provided a systematic protocol that permits ATP and others to compare similar yet diverse international public-private partnership programs for advanced technology development across their salient features. Features of particular interest included program eligibility requirements, the nature of funded research, technical scope, the selection process, and public-private financial arrangements. As Chang expressed it:

Without a lexicon with which to analyze and frame our understanding of these programs, our knowledge of such programs will remain superficial and evaluation and comparison of these programs will lack the necessary underpinnings. (p. 10)

If one country’s program succeeds and another fails, then that success may reflect their differences rather than their similarities.

International Programs

Another view of programs abroad and their comparison with ATP was provided by proceedings of an international conference on evaluation hosted by ATP in 1998. ¹⁰⁰ Bringing together evaluators from around the world, the conference had the theme of economic evaluation of science and technology programs in industrial countries. Fifty abstracts were submitted from government agencies and academic institutions from around the world, including France, Germany, Italy, Switzerland, Norway, Australia, Romania, Israel, the European Union, and China. Five abstracts describing programs similar to ATP were selected for full paper development and presentation at the conference—papers from Switzerland, Germany, Norway, the European Union, and Israel. Five additional evaluation methodology papers were selected that were commissioned by ATP.

Differences in evaluation methodologies were readily apparent in these papers. As noted by Dr. Phillipe Laredo of the Centre de Sociologies de l’Innovation, France, nearly all the U.S. conference papers emphasized the importance of spillovers while references to spillovers were completely absent from presentations from other countries. According to Laredo: “The former focuses more on producing figures (what return for the public investment) while the latter insists more on images (what changes in the innovation landscape). They might well be two corners of the same story....” ¹⁰¹, ¹⁰²

Testing ATP Models with Data from Other Programs

Several models developed for use by ATP were tested with data from other programs that had a longer history of operation. The results of those tests not only demonstrated the workings of the models, but also conditioned expectations about possible findings when the models could be applied to ATP data.

Using Japanese Data to Pilot Test an Analytical Framework

Mariko Sakakibara, University of California, Los Angeles, and Lee Branstetter, Columbia Business School and the NBER (National Bureau of Economic Research), developed a framework to measure the economic impact of ATP-funded research consortia using patent data. ¹⁰³ While they were able to work with U.S. patent data to some extent, they also used Japanese data as a statistical “testing ground.”

According to the researchers:

Japanese Government involvement in publicly supported research consortia dates back to the late 1950’s; examining this data will enable us to observe the long-run impact of consortia on patenting outcomes. (p. 27)

They further noted:

Results from Japanese data indicate that the effect of consortia on patenting outcomes tends to persist for relatively long periods of time. In fact, patenting in the targeted area seems to increase a bit after the cessation of the consortium, before leveling off again in later years. ...

For our purposes, the important point to keep in mind is that the effect of consortia can be quite long lasting. This suggests that our estimates of the impact of ATP-funded consortia, based on only four years of data, may underestimate the total impact of research consortia on patenting outcomes of the firms that were involved. (p. 28)

Using Israeli Data to Demonstrate Productivity Measurement

Zvi Griliches of Harvard University and NBER, in collaboration with Manual Trajtenberg of Tel Aviv University and NBER, and Haim Regev of Israel’s Central Bureau of Statistics developed an econometric approach to estimate the productivity impacts of ATP on private firms receiving funding. ¹⁰⁴ But ATP’s history was not sufficiently long to allow application of the model. To demonstrate their approach, the researchers investigated data requirements and tested the model with Israeli data from ATP counterpart programs. Their work is covered in more detail in Chapters 4 and 7.

Conditioning Expectations About Program Time Horizons

Stakeholder requests for empirically based measures of program impact soon after ATP began alerted program administrators that they needed to condition expectations about the timing of outputs, outcomes, and impacts. They needed to communicate that ATP’s larger benefits will take time; that technology creation, commercialization, and broad diffusion is a lengthy process. But they also needed to be more specific than simply stating that more time is required.

The question of appropriate time horizons affects not only evaluation but also project selection. When is a project too short term or too long term to fit ATP? This particular question was of keen interest to ATP. The notion that a project can have a too-short time horizon stems from an assumption that difficult technical problems will take time to solve. So, a too-short time horizon may mean that “low hanging fruit” projects—projects pursuing low risk technologies —have been selected for awards, undercutting a rationale for the program. On the other hand, a project whose anticipated benefits lie a number of decades in the future are deemed at too early a stage for ATP, because of the program’s emphasis on producing economic benefits through accelerated development and commercialization of technology.

Conceptual Time Path

Figure 4–8 shows a conceptual depiction of ATP’s time path that ATP staff has used to condition expectations about the program’s time horizon. The figure also lists the type of effects anticipated at each stage, conditioning expectations about what can be measured by evaluation efforts at different stages.

Figure 4–8. Conceptual Time Path for ATP’s Technology Diffusion and Expected Impacts

Economic impacts are depicted on the vertical scale and time on the horizontal scale. The lower curve, “benefits to awardees,” shows returns to the project innovators increasing over time as they commercialize or license their technology. The upper curve, “total economic benefits,” shows returns to the economy at large increasing as the technology diffuses into wider use and generates spillovers. The conceptual benefits curve starts above zero at the time of competition announcement, implying that there will be benefits from the technology project planning and formation of collaborations stimulated by the announcement. The “total economics benefits” curve is drawn more steeply as it begins to separate from the “benefits to awardees” curve toward the project end, signifying an expectation of increasing spillover benefits over time.

Source: Ruegg, “Assessment of the ATP,” 1999, p. 19.

Timeline Examples

In 1999, William Long, a consultant in economics affiliated with NBER—and assisted by ATP staff—documented the specific time paths for two of the first 38 completed ATP projects. ¹⁰⁵ One of the projects was in biotechnology, and the other was in computer software. Long found many of the same activities occurring in both timelines, but at very different times. The biotechnology project, which involved development of a medical technology as an outgrowth of university research, required a lengthy path of regulatory approval. In contrast, the software developer was able to enter into licensing agreements with other companies near the end of the project, and therefore, the timeline from research to commercialization was compressed.

The fact that both of these projects took on significant technical challenges, essentially stayed on their respective but different tracks, and met ATP’s expectations implied that there is considerable variation in acceptable time horizons among successful projects. The type of technology appeared to play a major role in the time required for commercialization and diffusion, as well as other factors such as business strategy, financing, regulatory requirements, and demand factors. These examples conditioned stakeholders to expect variation in timing among projects, even as the model shown in Figure 4–9 continued to provide general guidance. ¹⁰⁶

Testing Dominant Paradigms

Modeling underlying program theory helps administrators check the applicability of mainstream propositions before setting out on an evaluation design built upon those propositions. The abundance of existing theories and models can be deceptive, leading to uncritical acceptance of analytical assumptions and paradigms that do not fit. As understood by ATP, this caution does not lead to a rejection of mainstream approaches; rather it serves as a reminder to avoid premature analytical or methodological closure by quickly settling for dominant paradigms.

Productivity Impacts on Private Firms in Public Partnership Programs

Among the influences conditioning negative expectations about ATP’s impacts were previous studies of the impacts of government defense- and space-related R&D on firm productivity. Some of these studies suggested that the economic impact of government-funded research is lower than on private research or largely negative. ¹⁰⁷ The lower productivity of the defense side of large companies serving government defense and commercial markets has been put forward to support this proposition.

The question of whether government funding of private companies has an unavoidable depressing effect on the productivity of the firm’s R&D is important in determining the “net” impact of ATP’s support of private sector R&D activities. Formulation of the problem clearly relates to a central issue surrounding ATP’s operations: that is, whether the program affects the productivity as well as the total amount of R&D conducted by ATP awardees.

To test the feasibility of econometric estimation of productivity impacts on firms, Griliches, Trajtenberg, and Regev turned to Israeli counterpart programs as noted earlier. ¹⁰⁸ Israel, with its approximately 20 years of experience in wide-ranging efforts to develop and promote its high technology sectors, provided a test bed for assessing the productivity impacts of government support of private sector R&D.

Findings based on application of their model indicated that for the full period covered in the study, 1975–1994, government-supported R&D was “ not wasted in the Israeli economy and may even have had a higher rate of return than privately-financed R&D ” ¹⁰⁹ (italics in original). The authors expressed confidence in the thrust of their main findings, namely that the mechanisms used in allocating Israeli government funds to support firm-based R&D, “seem to be doing their work properly in most cases, and that the more they manage to ‘pick winners’ the better.” However, the authors explicitly noted that their estimates “should be treated cautiously;” the estimates of the coefficients for the grants variables are seen to be unduly high, pointing to potential problems in model specification, exclusion of relevant variables, or selection bias that originates in the way grant-receiving project or firms are chosen.

Other ATP-commissioned studies that bear on the topic of impacts of government sponsored R&D on firm productivity include work by Mariko Sakakibara, University of California-Los Angeles, and Lee Branstetter, Columbia Business School and the NBER. Their research found that participating in ATP consortia increased patenting in the funded areas above the level of patenting prior to the formation of the consortia. Another study that bears on firm productivity was carried out by Michael Darby and Lynne Zucker of UCLA and the NBER, and Andrew Wang, ATP economist. They point out that although there are arguments that government grants crowd out private R&D expenditures, they found evidence to the contrary for ATP. They concluded that ATP grants to private firms in fact increase the success of R&D in the recipient firms, noting that study results show these effects to be much more obvious for participant firms’ total patenting than for the direct results of the funded research projects.

Diffusion of New Technology

Achieving ATP’s ultimate goal of generating broadly based economic benefits requires technology diffusion. Models of technology diffusion abound. ¹¹⁰As with other relationships on which a sizeable body of research exists, the applicability of off-the-shelf-models in designing an appropriate evaluation methodology becomes an early design question. For example, among the immediate questions to be answered: (1) Are the same variables used in other studies the most highly significant or explanatory variables affecting the adoption of ATP-sponsored technologies? (2) Do early commercial planning and collaborative relationships with the downstream firms and actors involved in commercializing the new technology affect rates and levels of diffusion?

ATP’s Integrated Set of Strategies for Promoting Technology Diffusion

A NATO workshop designed to help policymakers in Central and Eastern European countries use technology transfer as a tool for transformation to a market economy provided ATP staff with an opportunity to review the program’s approach to technology diffusion. The resulting paper identified a set of specific strategies intended by ATP to promote technology diffusion. ¹¹¹ These strategies are summarized in Table 4–8.

Table 4–8. ATP Strategies Designed to Promote Technology Diffusion

Source: List of strategies summarized from Ruegg, “Assessment of the ATP,” 1999, p. 19.

The paper discussed ATP’s attention to selecting projects with structures intended to foster commercialization and diffusion of new technology and illustrated seven organizational structures based on ATP projects. ¹¹² The paper makes a logic-based argument that ATP projects have better diffusion prospects than they would in absence of the described set of integrated ATP strategies to promote diffusion, but does not prove it.

Analysis of Deployment Prospects for Selected ATP-Funded Technologies

In recognition of the complexity and difficulty of launching new technologies, and to further the early launch objective, ATP commissioned a team of researchers at ERIM, the University of Michigan’s Transportation Research Institute, and the Michigan Manufacturing Technology Center’s Performance Benchmarking Service to investigate the deployment prospects for a group of technologies funded under ATP’s focused program in Motor Vehicle Manufacturing Technology. ¹¹³ The researchers looked at technologies with potential for adoption by small and medium-sized manufacturing enterprises. They selected for detailed analysis the following three ATP-funded technologies: (1) agile precision sheet-metal stamping, (2) machine tool process monitoring diagnostic system, and (3) motor vehicle rapid toolmaker. These were also chosen for their potential application outside the automotive sector. The goal of the evaluation project was to identify ways to improve the prospects for broad deployment of ATP-funded technologies.

The researchers noted that all three of the ATP-funded projects used “traditional means” to encourage early use of the new technologies. They cited the presence of “lead users,” beta testing at customer sites, informational meetings with leading companies, technical interchange meetings, promotion of the technologies at trade shows and technical meetings, technology transfer workshops, technology demonstrations, posting information at websites, poster sessions, and industry association activities. They cited sharp differences among the projects in their use of technical publications as a communications media, with the differences attributed to differing strategies to protect intellectual property. The researchers also noted the lack of available market research data at the time of the study relevant to the three case technologies.

The study report emphasized how important it is for the industry project team and the ATP project management staff to understand the innovation system within which the technology is to be deployed. A detailed knowledge of this system is necessary for gauging the commercialization prospects of technologies proposed in a given area. Figure 4–9 depicts conceptually an innovation system based on social interaction. Mapping social relationships in an innovation system can be useful in facilitating technology deployment and in assessing impacts.

An example of a barrier to deployment of new technologies is inherent in the authors’ use of the term “competence-destroying innovations.” Resistance to having one’s competence destroyed by the appearance of a new technology can be expected, as can the need to develop new competencies in the adopting community. The authors suggested that studies to determine how a new technology may affect existing competencies, followed by remedial actions, such as awareness seminars, could improve the chances of successful deployment.

Figure 4–9. A Social Interaction View of Innovation

Source: Adopted by Przybylinski et al., Temporary Organizations for Collaborative R&D: Analyzing Deployment Prospects, 2000, from Havelock and Havelock.

The authors identified another potential barrier to deployment in the requirement for “gating,” or prerequisite, technologies that must be in place before the new technology can be adopted. As an example, they cited the need for sophisticated use of computer-aided design to be practiced in order for small- and mediumsized enterprises to benefit from one of the three new technologies. Concurrent engineering practice was another example of a gating technology identified by the study. Hence, identifying in advance any gating technologies and taking early action to enhance their use may help smooth the way for deployment of a new technology. The authors saw a potential role for NIST’s Manufacturing Extension Partnership program to collaborate with ATP in this regard.

University-Industry Roles and Relationships

Since the early 1980s, a major thrust of federal and state government innovation policy has been to foster cooperation and collaborations between U.S. firms and universities. Several ATP studies have examined collaboration between firms and universities in projects funded by ATP.

ATP’s authorizing language focused on the needs of U.S. firms and industries. Following this intent, ATP focused its early program design and selection criteria on variables and relationships deemed salient to industry, treating universities as supporting players to be involved in ATP projects as firms chose. But observing the behavior of firms that applied for ATP awards, it became apparent that collaborative relationships between the firms and universities were more important than suggested by the original program language. Over time it became evident from proposals submitted by firms to ATP that firms were choosing to include universities as R&D collaborators in major ways. But what is the role of universities in these projects, and how does their inclusion affect project outcomes? A dominant a priori answer is that companies turn to universities to help them plan and conduct very advanced research. Several evaluation studies have shed light on these questions, and offered a few surprises.

Bronwyn Hall, the University of California, Berkeley, Albert Link, University of North Carolina-Greensboro, and John Scott, Dartmouth College, used a sample survey of ATP projects to investigate university involvement and effects. ¹¹⁴ They concluded that projects with university involvement are more likely to be in areas of “new” science. They found—with the caveat that the sample was small—that projects with university involvement are likely to experience more difficulty and delay, but also are more likely not to be aborted prematurely. This finding seems consistent with the dominant paradigm that universities are helping companies take on difficult problems at the forefront of research and helping them ultimately succeed.

In a related study, Bruce Kogut and Michelle Gittelman, professors from the Wharton School and NYU’s Stern School of Business, investigated public-private partnering among U.S. biotechnology firms. Their research also produced findings related to university involvement in ATP projects, that is, “firms with weak inhouse research capabilities can strongly improve those capabilities through collaboration with university scientists.” ¹¹⁵ They concluded that “...scientists who both publish and patent are critical channels through which scientific knowledge is applied to the innovations of the firm,” ¹¹⁶ and that partnering with university scientists improves a firm’s ability to attract such research talent. Again, their findings support the dominant paradigm, broadening it to include the effects of university affiliation on the firm’s ability to attract top researchers.

A somewhat different, challenging view of the relationship between universities and industry in ATP research projects emerges from a study done by Julia Liebeskind, University of Southern California’s Marshall School of Business. ¹¹⁷ Pointing to what may be thought of as an inverted path of knowledge flow—not from university to industry, but from industry to university—the author states:

More generally, the ATP may wish to pay more attention to the university- industry connection as one path for diffusion of technologies. It is apparent from these case studies that many scientists in firms believed that the science they were practicing was far ahead of university science. If so, collaborations between industry and universities of ATP-funded research could be one way in which industry advances could be fed back into education in the future.

Thus, the research sponsored by ATP of university-industry relationships extends the predominant paradigm to include more, and richer roles and effects than may have been apparent on the surface.

Summary of Research Informing Underlying Program Theory

This chapter has drawn models, concepts, and findings from 25 reports and papers that together have advanced understanding of ATP, conditioned expectations about the program’s effects, and indicated ways to improve it. Implicit or explicit in this body of work are the major arguments used by economists to explain the rationale for ATP, including the view that enabling technologies tend to generate large spillovers; global economic competition is increasingly driven by technological advance; high-tech risks contribute to an R&D funding gap in the private sector; many advanced technological development projects require synergistic, multi-disciplinary, and multi-organizational collaborative efforts whose initiation may require an outside stimulus; and the nation’s capacity for economic competitiveness and prosperity ultimately depends on the health of its innovative capacity which public-private partnerships can strengthen.

Because of the central role of the spillover concept in ATP’s rationale, the program commissioned an early report on knowledge, market, and network spillovers—what they are, how they are generated, and how to increase them. Earlier studies showing that market spillovers tend to result from private firm innovation suggested both their potential importance and the fact that ATP would have to set a goal of generating higher-than-average spillovers to be successful as a public program.

To help explain the multi-dimensions of ATP to stakeholders, several models were developed. One model graphed ATP’s impact as a trajectory in three dimensions of innovation space: knowledge creation, private firm benefits, and spillovers, explaining why ATP funding is not corporate welfare. Another model depicted ATP’s impact occurring along two paths. A direct path, subject to greater influence by ATP, depicted accelerated technology development by U.S. firms. An indirect path of knowledge flows, the direction and timing of which is hard to predict, depicted ATP’s main impact through supporting enabling technologies and encouraging the sharing of non-proprietary knowledge. Another important early step was to convert the concepts that emerged from modeling and hypotheses to measurable variables that could serve as program metrics and to develop an implementation strategy for data collection.

A study of factors associated with the success or failure of joint ventures represented an effort to better understand the connection between one of ATP’s design features and related outcomes. Factors most important to success appear to include those that promote trust among project participants. The ATP’s emphasis on collaboration is only one of a number of program features that could be subjected to such study. Thus, the study of program dynamics appears a relatively under-researched area within the broader effort to model the ATP’s underlying theory and concepts.

Investigation of funding availability in the private sector for high-risk research and of private funding decisions in the face of risk provided evidence of a funding gap for high-risk research. Findings from two studies reinforce the view that there is a “Valley of Death” where insufficient funding for early stage, high-risk technologies exists despite large inflows of private venture funding. Further, findings support the view that this gap can be bridged at least in part by government without driving out private sector funding.

Study of private sector practices further suggests that the private sector has created no single, simple, or dominant “silver bullet” R&D project selection method that must be followed by a public sector program. Study findings point to multiple definitions of the “success” or “failure” of technical projects, of key differences between the private and public sectors in the appropriateness of using commercial viability as the only or primary measure of success for public sector technology development programs, and of the importance of spillovers as a major benefit of public-sector R&D programs. ¹¹⁸ Findings point to the interrelationship of technical and market risks, and to the relatively greater risks associated with market volatilities. Findings also suggest that linking R&D project selection to measures of market impact tends to increase the accuracy of forecasts about rates of return, hence, reducing market risk.

Studies of state technology programs have been useful in understanding the positioning of ATP relative to these other programs. Study findings suggest that ATP is complementary to, rather than competitive with state programs, and that innovating firms can and do benefit from both.

Studies of foreign counterpart programs have provided insights for ATP and have helped ATP meet its mandatory requirements to determine eligibility of foreignowned firms for ATP participation. In addition, programs in Japan and Israel, with their longer histories and larger databases, have served as testing grounds for researchers developing ATP-commissioned assessment models. The results of these trial applications have also provided valuable insights relevant to ATP, such as evidence that the impact of a partnership program on firm productivity can be strongly positive and will likely be understated unless data are collected over an extended period.

Conceptual modeling of ATP’s time horizon with delineation of types of outputs and outcomes expected to unfold over time has helped condition stakeholder expectations. Empirical studies of actual time horizons for different technologies have served as a reminder of the range of variation to be expected within the boundary of the general framework. Analysis and testing of dominant paradigms, rather than their unchallenged acceptance, has led ATP to develop more appropriate models and concepts for the program. For example, several studies have challenged the dominant view that government funding lowers private firm productivity.

Analysis of ATP’s attention to project structure and program features, which may increase the propensity of projects for commercial development and technology diffusion, has also challenged the paradigm that a “one-size-fits-all” approach to estimating rates of technology commercialization and diffusion is appropriate. Studies of university roles and relationships with firms in innovation projects have both supported and challenged the dominant paradigm that knowledge generated by projects flows from universities to industry, with one study pointing to the importance of the reverse order of flow, from industry to universities.

This body of work represents an important investment by ATP because it lays important pieces of the analytical and conceptual foundation of good program management and good evaluation practice. As Spender has aptly noted: “Appropriate theory is badly needed if we are to make better sense of our experience of these programs.” Clearly, the job of developing appropriate theory for partnership programs is not completed, but substantial progress has been made.

_____________________
60 An economic spillover is also known in economists’ jargon as a “positive externality.”

61 Adam B. Jaffe, Economic Analysis of Research Spillovers: Implications for the Advanced Technology Program, NIST GCR 97–708 (Gaithersburg, MD: National Institute of Standards and Technology, 1996).

62 J-C Spender, “Publicly Supported Non-Defense R&D: The U.S.A.’s Advanced Technology Program,” Science and Public Policy, February: 45–52, 1997.

63 Ibid., p. 45.

64 The implication of the diagram is that these are linear relationships. In practice, ATP’s evaluation design was predicated on an understanding that the above relationships were likely to be neither linear nor describable as two singular pathways; however, these simplifications were made for the purposes of exposition.

65 R. Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, NISTIR 6385 (Gaithersburg, MD: National Institute of Standards and Technology, 1999); also see Ruegg, “Assessment of the ATP,” 1999. be slower and less amenable to a deliberate program focus on commercialization of technology by U.S. companies.

66 R. Ruegg, “Delivering Public Benefits with Private-Sector Efficiency through the Advanced Technology Program.” In Charles W. Wessner, ed., The Advanced Technology Program: Assessing Outcomes (Washington, DC: National Academy Press, 2000).

67 Albert N. Link, “Measuring the Economic Impact of the Advanced Technology Program: A Planning Study,” unpublished report, 1992.

68 See, for example, U.S. Department of Commerce, Annual Performance Plan, “Science, Technology, and Information Performance Measures for the Advanced Technology Plan.” U.S. Department of Commerce, Washington, D.C. An annual performance plan is issued each fiscal year.

69 Jeffrey H. Dyer and Benjamin C. Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, GCR 00–803 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

70 To learn more about the workings of joint ventures, ATP also hosted a “lessonslearned” workshop with participants of funded joint ventures at NIST on May 22, 1996. Workshop proceedings are available on-line at ATP’s website (http://www.atp.nist.gov/alliance/best_p.htm).

71 Because of the relatively small sample size, their findings should be regarded as suggestive and tentative.

72 Edwin Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities , NIST GCR 99–780 (Gaithersburg, MD: National Institute of Standards and Technology, 1996).

73 Edwin J. Mansfield, J. Rapport, A. Romeo, S. Wagner, and G. Be Ardsley, “Social and Private Rates of Return from Industrial Innovation,” Quarterly Journal of Economics 91(2): 221—240, 1976.

74 Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996, p. 28.

75 Ibid., p. 2.

76 Ibid., p. 7.

77 Mansfield’s death unfortunately ended the project.

78 Paul Gompers and Josh Lerner, Capital Formation and Investment in Venture Markets: Implications for the Advanced Technology Program, NIST GCR 99–784 (Gaithersburg, MD: National Institute of Standards and Technology, 1999).

79 Ibid.

80 The authors’ focus was on financing of companies and not on the enabling nature of the technologies, that is, underinvestment owing to spillovers was largely outside the study’s sphere. “Early-stage” was used as a proxy for the type of projects of interest to ATP. It captures one aspect of ATP projects but is not necessarily synonymous with funding high-risk, enabling technologies.

81 L. M. Branscomb, Kenneth P. Morse, and Michael J. Roberts, Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, NIST GCR 00–787 (Gaithersburg, MD: National Institute of Standards and Technology, 2000). This study provided the basis for the subsequent publishing of L. M. Branscomb and Philip E. Auerswald, Taking Technical Risks: How Innovators, Executives, and Investors Manage High-Tech Risks (Cambridge, MA: MIT Press, 2001).

82 Branscomb et al., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000, p. 3.

83 Ibid., p. 1.

84 Ibid., p. 8.

85 Ibid., p. 15.

86 Ibid., p. 16.

87 Ibid., p. 23.

88 Ibid., p. 22.

89 Branscomb et al., Managing Technical Risk: Understanding Private Sector Decision Making on Early Stage Technology-Based Project, 2000.

90 Jenny C. Servo, Dawnbreaker Press, © Commercialization and Business Planning Guide for the Post-Award Period, NIST GCR–99–779 (Gaithersburg, MD: National Institute of Standards and Technology, 2000).

91 Scott Fosler, New Economic Role of the States (New York: Oxford University Press, 1988).

92 Peter Eisinger, The Rise of the Entrepreneurial State (Madison, WI: University of Wisconsin Press, 1988); Irwin Feller, “American State Governments as Models for National Science Policy,” Journal of Policy Analysis and Management 11:288–309, 1992.

93 Dan Berglund and Christopher Coburn, Partnerships: A Compendium of State and Federal Cooperative Technology Programs (Columbus, OH: Battelle Press, 1995).

94 Marsha R.B. Schachtel and Maryann P. Feldman, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. I, A Guide to State Business Assistance Programs for New Technology Creation and Commercialization, NIST GCR 00–788 (Gaithersburg, MD: National Institute of Standards and Technology, 2000); and Maryann P. Feldman, Maryellen R. Kelley, Joshua Schaff, and Gabriel Farkas, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, vol. 2, Case Studies of Technology Pioneering Startup Companies and Their Use of State and Federal Programs, NISTIR 6523 (Gaithersburg, MD: National Institute of Standards and Technology, 2000).

95 Schachtel and Feldman, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, 2000, p. 1.

96 Zvi Griliches, Haim Regev, and Manuel Trajtenberg, R&D Policy in Israel: Overview and Lessons for the ATP, Draft report, ATP, 2000.

97 Sec. 278n, (9)(A & B), P.L. 100–418, amended by P.L. 102–245.

98 All proposals must meet the test of being in the national interest of the United States.

99 See Connie K.N. Chang, “A New Lexicon and Framework for Analyzing the Internal Structures of the U.S. Advanced Technology Program and its Analogues Around the World,” Journal of Technology Transfer , 23(2): 67—73, 1998.

100 Richard N. Spivack, ed., Proceedings of an International Conference on the Economic Evaluation of Technological Change, NIST Special Publication 952 (Gaithersburg, MD: National Institute of Standards and Technology, 2001.) The conference co-chairs were Richard Spivack, ATP, and Lee Branstetter, then at University of California, Davis, and now at Columbia Business School and the NBER (National Bureau of Economic Research).

101 Philippe Laredo, Comments, in Richard N. Spivack, ed., Proceedings of an International Conference on the Economic Evaluation of Technological Change , NIST Special Publication 952 (Gaithersburg, MD: National Institute of Standards and Technology, 2001) pp. 158–159.

102 These differences in evaluation methodology have been noted by both U.S. and European evaluators and policy makers for a number of national R&D programs, not only those closely related to ATP. Differences in approaches to evaluation reflect differences in the political, economic, and academic environment, and differences in the way evaluation is organized in the various countries. As expressed by Feller, describing the relatively more decentralized character of the U.S. evaluation system, “Multiple sponsors fund multiple researchers located in multiple institutions; the result is a diverse, at times competitive evaluation marketplace, in keeping with the characteristics of the U.S. political and academic systems. Methodological orthodoxy, thankfully, is impossible to establish.” (Irwin Feller, “The Academic Policy Analyst as Reporter: The Who, What, and How of Evaluating Science and Technology Programs,” in Philip Shapira and S. Kuhlman, eds., Learning from Science and Technology Policy Evaluation (London: Edward Elgar, 2001)).

103 Mariko Sakakibara and Lee Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, NIST GCR 02–830 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

104 Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, Draft report, ATP, 2000.

105 William F. Long, Performance of Completed Projects, Status Report 1, NIST Special Publication 950–1 (Gaithersburg, MD: National Institute of Standards and Technology, 1999), pp. 5–7.

106 More recently, ATP’s Business Reporting System survey data have provided more empirical data on timelines and how they differ among technologies. See Jeanne Powell and Francisco Moris, Different Timelines for Different Technologies, NISTIR 6917 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

107 See, for example, Z. Griliches, “Productivity, R&D, and Basic Research at the Firm Level in the 1970’s,” American Economic Review, 76(1), 141–54, 1986; Z. Griliches and Frank Lichtenberg, “R&D and Productivity at the Industry Level: Is There Still a Relationship?” in Z. Griliches, ed., R&D, Patents and Productivity (Chicago: University of Chicago Press, 1984), pp 465–496; David Levy and Nestor Terleckyj, “Effects of Government R&D on Private R&D Investment and Productivity: A Macroeconomic Analysis,” The Bell Journal of Economics, 14(2): 551–561, 1983.

108 The Griliches et al., study indicated that it was not possible to obtain Israeli data that would have permitted assessment of whether or not government supported R&D in Israel generated spillovers. (“While the...aspect of ...externalities, may be the most important in evaluating the success of such support programs, our data will not permit us to pursue it,” Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, Draft report, ATP, 2000).

109 Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, Draft report, ATP, 2000, p. 48.

110 See, for example, M. Karshenas and P. Stoneman, “Technological Diffusion.” In P. Stoneman, ed., Handbook of the Economics of Innovation and Technological Change (Oxford, UK: Blackwell, 1965), pp. 265–297.

111 See R. Ruegg, Advanced Technology Program’s Approach to Technology Diffusion, 1999.

112 Ibid., p. 15.

113 Stanley M. Przybylinski, Sean McAlinden, and Dan Luria, Temporary Organizations for Collaborative R&D: Analyzing Deployment Prospects, Draft report, ATP, 2000.

114 Bronwyn H. Hall, Albert N. Link, and John T. Scott, Universities as Research Partners, NIST GCR 02–829 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

115 Bruce Kogut and Michelle Gittelman, Public-Private Partnering and Innovation Performance among US Biotechnology Firms, Draft report, ATP, 2000.

116 Ibid., p. 34.

117 Julia Porter Liebeskind, Study of the Management of Intellectual Property in ATPGrantee Firms, Draft report, ATP, 2000.

118 Spender, “Publicly Supported Non-Defense R&D: The U.S.A.’s Advanced Technology Program,” 1997, p. 51, referencing Adams, Stoffaes, and Graham.

Return to Table of Contents or go to Chapter 5: Survey Method.

Date created: July 22, 2004
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)