A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 4:  Modeling and Informing Underlying Program Theory

As discussed in Chapter 2, the modeling and informing underlying program theory is an essential prelude to program operations, an early stage complement to other management and evaluation efforts, a building block in the design of a long-term comprehensive evaluation program, and an ongoing source of continuous organizational learning. Clarifying and validating a program’s underlying concepts and the analytical linkages among its various elements is an important part of an agency’s overall program evaluation strategy. Yet, pressures for quick responses to questions about program activities can lead program administrators to neglect program theory and the investigation of program dynamics.

At ATP, modeling and informing underlying program theory has yielded information critical to the program’s survival, shape, and success, and has become a mainstay of ATP’s evaluative program. Identifying the characteristics of multiple, complex, causal paths has been an important part of ATP’s evaluation plan because program modeling strategy involves influencing both the immediate and proximate determinants of firm and industry behavior. The combination of external challenges and internal commitment to documentation of results has produced a program noteworthy among federal government programs in the level of attention it has devoted to understanding and documenting the program’s effects.

This chapter details ATP’s use of analytical and conceptual modeling to explore basic concepts and of underlying program relationships, to condition expectations and set benchmarks for performance, and to refine dominant theoretical paradigms. Unavoidably, the numerous relevant concepts and subtopics drawn from multiple sources add complexity to the treatment. To assist the reader, the major themes and the reports and papers used to amplify each theme are listed in Table 4–1, presented in the order discussed in the chapter, and with the particular contribution to underlying concepts and theory noted. The primary purpose of several of the documents cited here was not to explicate underlying program theory, yet they are included because they also made significant contributions to understanding ATP’s workings. Table 4–1 is intended as a quick reference and roadmap through the chapter.

Table 4–1. Twenty-Two Studies and Papers Modeling and

*Note: Three of the studies are listed more than once because they address multiple concepts.

Concepts, Models, Metrics, and Paths Connecting Program Activities to Intended Impacts

Recognizing that it fell between the conventional poles of federal support of basic research on the one hand, and of mission-oriented basic and applied research on the other, ATP quickly sought to clarify its basic concepts, to help explain how it works, and to develop improved models for analysis. A variety of seminars, roundtables, staff papers, presentations, and commissioned studies were used to flesh out concepts, models, and pathways of the program, and point toward the best evaluation methods.

Economic Spillovers

The concept of spillovers is central to ATP and implicit in the argument that ATP will help address under-investment in generic technologies by the marketplace. An “economic spillover” ⁶⁰ is the excess return to society of an investment over the private return captured by the investing firms. The presence of particularly large spillovers may cause private firms to invest less than is socially optimal because too much of the benefit escapes them. Spillovers provide a primary justification for public intervention to increase investment in R&D. A closely related concept is “inappropriability,” which refers to the inability of an investor to capture, or “appropriate,” the fruits of his or her investment.

To delve deeper into the subject of spillovers and their implications for the program, ATP commissioned Adam Jaffe, economics professor at Brandeis University and member of the National Bureau of Economic Research (NBER), to prepare an economic analysis of research spillovers. ⁶¹ Noting that spillovers have been of interest to economists for several hundred years, Jaffe pointed out that R&D activities of private firms have been shown to generate spillover benefits; hence, ATP should be expected not only to fund R&D that generates spillover benefits, but R&D that yields higher-than-average spillovers.

Jaffe’s work identified three different sources of spillovers relevant to ATP— knowledge spillovers, market spillovers, and network spillovers—and noted that the three interact synergistically to increase their combined effect. Table 4–2 defines each source of spillovers and gives examples of how each occurs. Figure 4–1 illustrates how knowledge and market spillovers interact and may lead to a gap between social and private returns.

Table 4–2. Classification of Spillovers

Source: Summarized from Jaffe, Economic Analysis of Research Spillovers, 1996.

Figure 4–1. Private and Social Returns to R&D: Pure Market Spillover, Plus Pure Knowledge Spillover, Plus Interaction of the Two

Firm 1 invests in R&D, generating new knowledge that it uses to improve its products or lower its production costs. Assuming the firm successfully commercializes the results, market competition causes the value of some of firm 1’s improvements to be captured by its customers in the form of lower prices or higher quality. This effect alone would cause a spillover gap equal to the customer benefit. But the figure shows other effects. The first downward pointing arrow indicates that knowledge spillovers flow from firm 1’s knowledge base to other firms through disembodied outputs such as papers and patents. The second downward pointing arrow indicates that knowledge also passes from firm 1 to other firms through research results embodied in its new commercial products and processes. The third arrow, which points upward, indicates that at least some of the firms benefiting from the knowledge spillovers are competitors of firm 1, who then introduce cheaper or better products into firm 1’s markets, taking some of its profits and creating some additional customer benefits. Meanwhile, these other firms may also introduce improved or lower cost products and process into their own markets, resulting in firm profits and customer benefits. As Jaffe observes, “...the combination of knowledge spillover with competitive interaction increases the spillover gap both by raising the social return and lowering the private return.” (p. 17)

Source: Jaffe, Economic Analysis of Research Spillovers, 1996.

In drawing the implications of spillovers for ATP, Jaffe identified factors that he saw associated with higher spillovers—a desirable feature from ATP’s point of view. Table 4–3 lists some of these factors. According to Jaffe, ATP could improve its ability to select projects with greater-than-average spillover potential by acting on this information. This study gave ATP a model and specific guidance for increasing its effectiveness.

Table 4–3. Factors Increasing the Likelihood of Spillovers

Source: Compiled from information presented in Jaffe, Economic Analysis of Research Spillovers, 1996, pp. 42–44.

Descriptive Models of ATP

Models have more than analytical value; they can also help communicate a program to stakeholders. ATP staff developed several models to help explain the program’s mission to a diverse audience. These models were used to explain and discuss the program in settings such as policy forums, internal strategic planning sessions, and ATP workshops.

The Three Dimensions of ATP’s Mission

Modeling by J-C Spender, UK Open University Business School, helped ATP solidify its internal perspective and explain its dynamics. ⁶² The three dimensions shown in Figure 4–2 embody three main ideas that underlay ATP’s enabling legislation: scientific and technical knowledge gains (axis labeled Technical Knowledge), commercial gains and improved competitiveness of U.S. companies (axis labeled Private Returns), and broadly based spillover benefits to the nation at large (axis labeled Public Gains). The figure is a conceptual illustration, not intended for quantification.

In describing the program, Spender, the diagram’s author, speaks of ATP’s goal as “promoting trajectories through an innovation space formed by partnership between the USA’s three institutionally distinct modes of scientific and technological innovation: scientific research; private enterprise; and, public-sector management of society’s public goods.” ⁶³

Spender’s framework also has applicability in addressing critics’ complaint that ATP represented “corporate welfare.” According to Spender, charging that ATP is “corporate welfare” is to mistake, with reference to Figure 4–2, the point P for the point Q:

Critics are correct in the sense that market-driven firms are being subsidized and are therefore benefiting, but off the mark in the sense that the point P is incidental. It indicates only the agent’s incentive to act on ATP’s behalf, not the social result of its activity. Ultimately the project’s success must be measured in terms of the point Q (p. 49).

Figure 4–2. Three Dimensions of ATP and the “Journey to Q”

In concept, technical progress is measured in the diagram along the axis labeled Technical Knowledge, net benefits to the awardee from commercial activity along the axis labeled Private Returns, and net spillover benefits to others along the axis labeled Public Gain. The dimensions’ orthogonality illustrates the independence of these outcomes (i.e., technological advance does not necessarily lead to economic activity, and commercial activity does not necessarily lead to public interests). The point Q depicts a desirable outcome for ATP—the end point of a project trajectory producing a combination of knowledge gains and net benefits to the awardee that are exceeded by large net spillover benefits to others in the economy.

Source: Spender, “Publicly Supported Non-Defense R&D: The U.S.A.’s Advanced Technology Program,” 1997.

In addition to being useful in explaining the program’s design to external constituencies, Spender’s model also helped to develop a common perspective among ATP’s functional staff: primarily scientists and technologists, business specialists, and economists. In the early to mid-1990s, combining staff from these fields in a single unit was still relatively new in business and government, and there was a distinct learning curve for the staff. As Spender puts it:

ATP draws the participants into a common framework even if the innovative activity may not be comprehensible to those not institutionalized into the appropriate professional environment. Business experts are unlikely to appreciate the niceties of the research scientist’s problems or respect the professional judgments required, especially where the levels of technical risk are high. Likewise, even though ATP seeks low-risk commercialization and public-goods effects, the complexities and subtleties of the social and economic works in which these arise are often puzzling to scientists. (p. 50)

ATP’s Two Paths to Impact

Looking for a more succinct way to explain the program to diverse audiences, and specifically to provide context for examples of project impacts, ATP began to use the “two-path” diagram reproduced in Figure 4–3, which depicts direct and indirect paths to program impact. ⁶⁴ The “direct path” represents the commercialization route of the award-recipient and its close collaborators to capture private benefits and generate market spillovers, as well as possible knowledge spillovers as others reverse engineer their products and processes. The “indirect path” represents the main route to knowledge spillovers: award-recipient publications, presentations, interactions, patents, and mobility of researchers among organizations, allowing others to gain knowledge from the funded research without paying for it. If these other organizations use the knowledge obtained for economic gain, benefits from knowledge spillovers result. ⁶⁵

The direct path has special significance because it allows ATP directly to encourage U.S. businesses to accelerate development, commercialization, and use of new technologies. The indirect path may ultimately prove to be even more important than the direct path in its generation of benefits, but it tends to ⁶⁶ Nevertheless, ATP can foster the generation of knowledge spillovers by selecting generic technologies applicable to many firms both upstream and downstream and by supporting publishing, patenting, collaborative activities, and other activities through which knowledge is diffused.

The diagram highlights the extension of ATP’s role beyond that of technical knowledge generator/disseminator. By engaging the activities of for-profit U.S. firms and requiring them to plan up-front for commercialization and establishment of a pathway to early application of the technology, ATP seeks to meet its mandate to generate accelerated economic benefits and improved competitiveness of U.S. firms in international markets.

Planning Performance Metrics for ATP

An important component of ATP’s evaluation program has been to convert the concepts that emerged from modeling its underlying theory into variables amenable to measurement/quantification. In an early effort along these lines, ATP supported the work of Albert Link of the University of North Carolina, Greensboro, to recommend performance metrics for ATP. ⁶⁷ Link also provided advice on developing an implementation strategy for compiling the metrics.

Table 4–4 summarizes Link’s recommended set of measures and timetable for collecting them. The nuts-and-bolts data collection plan proved useful for ATP as it sought to implement its evaluation activities quickly. In a very practical way, Link’s plan informed the theory and pathways of program effect.

Figure 4–3. Direct and Indirect Paths to ATP’s Impacts

If a project’s participating companies successfully take the technology into commercialization (direct path), this is expected to speed the introduction of the technology by U.S. companies. If participants falter in the commercial phase, then they may nevertheless leave behind publications, patents, informed workers, and other means of transferring the knowledge to others who may take it forward. If activities along both paths are successful, then the gains can be even greater, because the social return of the project is the net impact from the combination of the two paths and their interactions.

Source: Ruegg, “Delivering Public Benefits with Private-Sector Efficiency through the Advanced Technology Program,” 2000.

Table 4–4. Recommended Metrics for ATP and Timing of Data Collection

Source: Link, “Measuring the Economic Impact of the Advanced Technology Program: A Planning Study,” 1992, p. 30.

ATP routinely collected much of the data listed in Table 4–4. In the mid-1990s, ATP expanded and revised its data collection efforts to be more responsive to Government Performance and Results Act of 1993 (GPRA) reporting requirements and stakeholder questions. ⁶⁸

Understanding Relationships Between Program Design Features and Outcomes

As noted, relationships between program design features and program outcomes are often sketchy. The discovery of systematic differences in the effectiveness of design-feature variations could improve funding decisions. For example, ATP’s enabling legislation emphasizes the role of research joint ventures in achieving ATP’s mission, but does not suggest which types of partnering arrangements may be most conducive to high levels of program performance. A better understanding of what makes joint ventures work could be useful in guiding ATP’s efforts. To this end, ATP supported a study by Jeffrey Dyer, Brigham Young University, and Benjamin Powell, University of Pennsylvania, to investigate factors that increase or decrease the likelihood of success of R&D collaborations. ⁶⁹, ⁷⁰

Dyer and Powell conducted semi-structured interviews with companies participating in 18 ATP-funded joint ventures developing technologies with application in the automotive industry. Their interview discussions centered on the questions listed in Table 4–5.

They found that greater knowledge sharing and more effective coordination among participants characterized the more successful joint ventures. They identified several factors that influence the extent to which participants share knowledge, and that influence the costs of coordinating the venture’s activities. ⁷¹ Their diagram showing the key factors that participants said influenced the degree of success of their joint ventures, the direction and nature of impact, and the linkages from these factors to measures of outcome success is shown in Chapter 9’s report on findings, Figure 9–4. Understanding better how the program contributes to collaborative success and which factors can make or break a joint venture are important for ATP administrators and program managers. For instance, Dyer and Powell documented the importance of face-to-face meetings of participants to build trust, and the critical role of trust in collaborative success. As standard procedure in the contracting process, ATP staff must approve proposed travel budgets for which ATP funds are to be used. So, without an understanding of this underlying success factor, well-meaning staff members could easily make what they intend as “efficient” budgetary decisions (e.g., refusing to approve travel costs for frequent face-to-face meetings among joint venture participants), which may have a negative effect on the project’s success.

Table 4–5. Interview Questions Investigating Determinants of Success of Joint Ventures

Source: Dyer and Powell, Determinants of Success in ATP-Sponsored R&D Joint Ventures: A Preliminary Analysis Based on 18 Automobile Manufacturing Projects, 2002.

Conditioning Expectations through Studies of Private-Sector Behavior

Modeling private-sector decisions and outcomes has helped define expectations of what the program can achieve and helped establish benchmarks for ATP’s performance, especially in shaping realistic expectations of the technical and economic successes of a firm’s R&D portfolio. Also, of use for ATP has been knowledge about private-sector tools for managing risk, as well as analysis of private-sector decisions in the face of risk, and evolving relationships among firms, universities, and other organizations. A better understanding of barriers to private investment in early stage, science-based innovations sheds light on ATP’s value-added role in financing of R&D projects.

Social Rates of Return from Business R&D

Early in its evaluation program, ATP explored the applicability of a widely used model developed in the 1970’s by Edwin Mansfield, University of Pennsylvania, to measure market spillovers of R&D. ⁷² ATP also explored the implications of Mansfield’s earlier application of his model to estimate private and social returns from new products or processes when private-sector innovators commercialize their technologies. ⁷³ As Mansfield writes:

From the point of view of the ATP program, the gap between social and private rates of return from investments in new technology is of central importance. After all, a major rationale for the ATP program is that some R&D projects have social rates of return far in excess of private rates of return. (pp. 26–27)

Mansfield’s Model

Mansfield measured resulting market spillovers in terms of “consumer surplus.” In Mansfield’s approach, the social benefits from an innovation are measured by the profits of the innovator from the innovation plus the benefits to consumers following reduction in the price of the good due to the innovation. ⁷⁴ Figure 4–4 illustrates his basic model.

Figure 4–4. Social Benefits from Product Innovation that Reduces Costs of Industries Using It

Assume that the supply curve was S1 before the innovation, and the price charged by the industry was P1. After the introduction of the innovation, the supply curve is S2, and the price charged is P2. The social benefits from the innovation are measured as the sum of the two shaded areas shown in Figure 4–4. The top shaded area is the consumer surplus due to the lower price, P2 rather than P1. The bottom-shaded area in Figure 4–4 is an estimate of the additional resource savings from the innovation. A resource savings results, leading to a corresponding increase in output elsewhere in the economy, because the resource costs of producing the good after the innovation are P2 Q2, minus the profits (r) the innovator receives from the innovation. The amount r is merely a transfer from the producers of the good using the innovation to the innovator. On net, there is consumer surplus from the price reduction and a resource savings amounting to the profits of the innovator. (Caveats are given by Mansfield in his report.)

Source: Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996, p. 20.

Mansfield’s findings on the characteristics of industrial R&D decision making are drawn from interviews with industrial R&D officials. The interviews point to the difficulty that even the most sophisticated firms have in forecasting the private returns from company-financed R&D projects. Despite the use of and search for formal decision-making models such as scoring systems, programming techniques, and other quantitative decision analysis techniques, “available studies,” according to Mansfield, “indicate that few firms, if any, are confident of their forecasts of particular R&D projects.” ⁷⁵ The reasons for this limited ability to accurately forecast rates of return relate to the inherent uncertainties of R&D projects, and to the difficulty that firms have had forecasting development cost and time, the probability of success, and the profitability of new products or processes. According to Mansfield’s analysis, firms face an additional difficulty in “forecasting how R&D, if successful, will be utilized.” ⁷⁶ Figure 4–5 illustrates that there is difficulty in trying to forecast private returns from investments in new technology early in the development process, and that the accuracy improves over time. The figure shows the extent of the forecasting errors within a single company for 57 new processes or products as a function of the number of elapsed years after their development. Shortly after the new processes or products were developed, the profit forecasting errors were high. As time passed, estimates of discounted profits were revised each year to the point that they became very close to actual discounted profits. The errors were eliminated through revision over time, but the error rate in the initial estimates was not found to improve over time.

Application of Mansfield’s Model to ATP

Several factors limit application of Mansfield’s model to ATP. Mansfield applied his model to single products, whereas ATP wanted to apply it to technology platforms generating multiple products. He used his model to capture only market spillovers, whereas ATP wanted to capture knowledge and network spillovers in addition to market spillovers. Finally, in estimating the model, Mansfield applied his model using many years of historical data, whereas ATP technologies were in the early stages of commercialization.

Figure 4–5. Reducing Inaccuracies in Profitability Estimates Over Time

^a Computed as the proportion of product/process cases for which the ratio of the profit forecasts made in each successive year after development to actual discounted profits is equal to or greater than 2.0 or less than or equal to 0.5.

Source: Mansfield, Estimating Social and Private Returns from Innovations Based on the Advanced Technology Program: Problems and Opportunities, 1996, p. 16.

To overcome these limitations, Mansfield recommended applying his model multiple times to capture market spillovers of enabling technologies with multiple applications. To meet concerns about the large forecasting errors early in the life of a new product and the fact that these errors tend to diminish rapidly as time passes, Mansfield recommended repeating and updating a set of case studies several times to see how the results changed over time. The commissioned a set of cases with Mansfield to be done according to his recommendations, recognizing that supplemental techniques would be needed to capture knowledge spillovers omitted from Mansfield’s model. ⁷⁷

Other case studies of ATP projects have followed Mansfield’s basic approach. See, for example, the calculation of social and private rates of return on medical technologies performed by Research Triangle Institute, discussed in Chapter 6.

Implications of Mansfield’s Model for ATP

Mansfield’s model provided a starting point for modeling the social and private rates of return of ATP-funded projects, and the empirical results from his work provided an initial benchmark for ATP’s assessment of the effects of its awards. Mansfield’s work, conducted in the late 1970s, included estimates of the private and social rates of return of 17 private-sector innovations. In this study he collected data on project costs, revenues, and profits. He then collected similar data from other firms in the same industry for similar products and processes they introduced, data from firms purchasing or licensing the new products or processes, and data from final users of consumer goods. From these data he calculated the private and social rates of return on the innovations. Of particular interest to ATP, Mansfield found significant market spillover benefits to consumers, even when the private returns were low or negative. The results became well known among economists, and have conditioned general expectation about the relationship between social and private rates of returns from technological innovations.

The results, in effect, “up the ante” for ATP; funding projects that yield positive market spillovers would not be sufficient for program success because “routine” private-sector innovation could do that. The existence of positive market spillovers from private-sector innovation underlies Jaffe’s observation, as noted above, that ATP would have to fund projects that yield higher-thannormal spillover effects if it were to add value as a public-sector program. This observation had important programmatic implications for ATP, because it led to a search for innovative projects with the potential to create substantial social benefits while meeting private-sector criteria necessary to gain participation by private firms.

Adequacy of Private Investment

The 1980s and 1990s saw a substantial increase in venture capital funds and investment funds available from “angel” investors. The larger amounts of venture funds were largely the result of a change in law that no longer prohibited institutional investors from venture investing. ⁷⁸ The larger amounts of angel funding were attributable in part to the dramatic increase in personal wealth in the United States over the same period. Table 4–6 traces the amount of venture capital raised each year from 1977 through 1995, as well as the amount of venture funding going to early stage investments. Nearly three times as much went to early stage investments in 1995 as in 1977, though at the time of preparing this report there has been a cyclical contraction in venture funding.

The increase in private sector supply of venture capital funding was one of the facts cited to challenge the need for government funding of high-risk technological innovations, and thus ATP. To explore this issue, ATP commissioned an inquiry into the supply of venture capital to fund enabling, high-risk technologies.

Paul Gompers and Josh Lerner, both professors at Harvard University Business School and members of the NBER, conducted the study, investigating the trends and patterns of venture capital and angel funding available to small innovative firms. ⁷⁹ Gompers and Lerner also conducted seven case studies of small R&Dintensive firms funded by ATP to determine why those firms needed ATP funding and the role the funding played. Several of their findings help to explain why, despite the large increase in private venture funds, projects of the type targeted by ATP can go wanting. ⁸⁰

Putting the supply of venture capital in perspective, the researchers found that despite the increases in venture capital supply, less than one tenth of 1% of business startups annually have received venture financing in recent years. They described lemming-like behavior on the part of many venture capitalists, leading to a concentration of investments in “hot” technical areas, while other areas attract little or no venture capital. They identified geographical concentration of the bulk of the venture capital, noting that companies in many parts of the nation receive little or no venture capital. They suggested that the shift in the source of venture capital away from the individual investor and toward the institutional investor means that there is a greater preference for less risky R&D and shorter time horizons for realizing returns on these investments. They suggested that investors, for management purposes, have tended to increase the size of individual investments, rather than increase the number of investments in proportion to the growth of funds available. Hence, a doubling in the amount of venture funds available does not result in a doubling in the number of projects funded, but rather an increase in the size of the average project, as a fund manager can better stretch limited managerial resources over a portfolio with a higher dollar value than one that contains more projects. From their case studies, described further in Chapter 6, the authors concluded that each of the companies examined struggled to obtain funding to undertake its innovative research and was unable to secure sufficient funding from private sources.

Table 4–6. Volume of Venture Capital Activity

N/A = Data not available.

Source: Gompers and Lerner, Capital Formation and Investment in Venture Markets: Implications for the Advanced Technology Program, 1999, p. 17.

Private Firm R&D Decisions in the Face of Risk

To define the role technical risk plays in the financing decisions of firms and to explore further the adequacy of funding for developing high-risk, enabling technologies, ATP commissioned Harvard’s John F. Kennedy School of Government, in collaboration with MIT’s Sloan School of Management and the Harvard Business School, to conduct a broad ranging study. ⁸¹The Harvard-MIT Project on Managing Technical Risk examined barriers to private investment in earlystage, high-risk technology development projects. The study’s objective was to assist ATP to “better identify projects that would not be pursued or would be pursued less vigorously without ATP support and at the same time are likely to lead to commercial success—with broad public benefits—with that support.” ⁸²

The Harvard-MIT study drew primarily on expert assessments offered by industrial, academic, financial, and government representatives participating in a series of workshops held between 1999 and 2001. The workshops were organized around discussions based on a set of papers commissioned by academic participants and practitioners.

The study, in part, updated Mansfield’s summary of the techniques, behaviors, and perceptions of R&D managers toward technical and economic risk. The Harvard-MIT project examined the institutional, behavioral, financial, and nonfinancial barriers that produced inadequate incentives for entrepreneurs, venture capitalists, and corporations “to undertake some varieties of early-stage, high-risk technology development projects that have potential to generate radically new products and processes.” ⁸³

Managing Risk

Several themes about risk and its effect on financing technology emerged from the Harvard-MIT workshops. Industry R&D participants emphasized the art of quantifying technical risk. Further, although participants noted the existence of numerous well-established methodologies for assessing technical risk, their consensus was that none of these methods was very successful. ⁸⁴ Still, they agreed that understanding the sources of risk and dealing with them systematically is important.

Participants expressed varying perspectives on the ability to separate technical risk from market risk. As posed by the project’s report, “In a radical technical innovation, can one expect to define product and process specifications, then engage in research that is sufficient to reduce technical uncertainties to an acceptable level?” ⁸⁵

The problem here, as described by several industrial representatives, is that the desired market “specifications” about the characteristics of the intended product can be unstable. Consumer requirements may change during the project. Independently, the performance of the technology may evolve differently than predicted. “Those differences require an adjustment in the specifications, which in turn requires that market estimates be adjusted, which in turn may suggest a further adjustment in product specifications.” ⁸⁶

Institutional Factors

As part of the Harvard-MIT study, a commissioned paper by Scott Shane, University of Maryland, tested the proposition that newly created firms are “a particularly appropriate institutional form within which to make success of radical, science-based innovations,” ⁸⁷ while a commissioned paper by James McGroddy found that “a large firm with deep technical roots has some advantages over the startup with limited resources, in that the former can incubate the technology for several years before bringing it to market, thus reducing substantially the uncertainties surrounding the technical challenges.” ⁸⁸ Generally, the Harvard-MIT study highlighted an increased inter-firm and inter-sector dependence of purchasers and suppliers of both R&D and goods on one another.

The findings and assessments presented in the Harvard-MIT study suggest that the process of commercializing promising technical concepts involves more than public sector funding of pre-commercial R&D. Rather, “institutional” factors as much as “economic” factors may hinder private investment in early stage, sciencebased innovations. This finding poses new questions and program design issues for ATP. Thus, according to the study:

...major changes are transforming the institutional structure of the high-tech industrial economy. Large corporations are increasingly focusing on their role as system integrators, low-cost producers, and distributors and marketers internationally, while outsourcing much of their innovation to mid-size and smaller, technically specialized firms in their supply chain. Where will those small-to-medium size firms get their insights into the art of the possible from new science, if not from the large firms they serve? Is this another reason for public programs like ATP? (p. 6)

“Valley of Death”

The Harvard-MIT study also addressed the adequacy of private-sector funding of high-risk R&D projects. Whereas the Gompers-Lerner study addressed the overall availability of venture capital, the Harvard-MIT study cited a striking degree of general agreement among technical entrepreneurs, high-tech business managers, and venture capital investors about the existence of a “Valley of Death”—a significant gap between federally funded basic research and industry-funded applied research and development.

According to the Harvard-MIT study: ⁸⁹

Congress appears to have intended the ATP to be a mechanism for addressing the “Valley of Death” area of research, which has been described by Charles Vest, president of MIT, as “mid-level” research, and by Branscomb as “basic technology research.”

If such a gap is real, the fact that growth of commercial R&D is outstripping the growth of government-funded [R&D] only means that the gap may be growing, not shrinking, as public funding of early stage research fails to keep pace with commercial development. (p. 49)

Positioning ATP in Capital Markets

There are a number of federal programs that provide grants, loans, and other forms of assistance to U.S. firms. One of the challenges of a program like ATP is to convey how it differs from other programs and the circumstances under which it is a suitable or unsuitable funding source for a given organization. This communication necessarily takes place with busy CEOs, CFOs, and company researchers against a cacophony of background noise. Clarity is critical.

Figure 4–6 reproduces a framework that locates ATP in the broader financing landscape. ⁹⁰ It is part of a planning guide prepared for use by ATP award recipients in the post-award period, but it also serves as a broader outreach tool for ATP.

The guide, an important tool of communication with ATP’s direct stakeholders, models complex business strategies for the technology platforms envisioned as downstream results of the R&D projects. Figure 4–7 illustrates ATP’s goal to foster creation of technology platforms that can spawn multiple business opportunities. The guide also covers teaming arrangements, and many other concepts critical to the awardees achieving not only business success, but also project success.

Figure 4–6. Roadmap of Financing Options

The figure shows five categories of financing in a financing roadmap. As indicated by the arrow and side note, ATP awardees are seeking an R&D partnership. Federal, state, and private sources provide financing opportunities through partnerships. The figure distinguishes ATP’s role from the other partnership opportunities. The figure also points to alternative sources of financing that award recipients can attempt to access as the ATP funding is drawn down. The Commercialization and Business Planning Guide positions ATP in the financing landscape and assists startup companies who need help with the ins and outs of seeking and obtaining financing from different sources.

Source: Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 20.

Conditioning Expectations through Studies of Other Public Sector Programs

With new programs come new expectations, and defining reasonable expectations is critical to a new program's success. Exmining state programs and counterpart programs abroad has helped condition general expectations about ATP.

Figure 4–7. Strategic Mind Map for a Rich Technology Platform

The figure shows an example of a company that has developed a portfolio of intellectual property to protect its process for creating a new material as well as the devices used to produce the material. The figure illustrates partitioning of multifaceted opportunities, taking into account downstream and upstream potential applications, intellectual property positioning, and economic factors. It shows multiple licensing opportunities (i.e., to license a process to produce material, to license production equipment, and to license in application-specific product market) and multiple application areas.

Source: Servo, Commercialization and Business Planning Guide for the Post-Award Period, 2000, p. 53.

Interface of ATP with State Programs

In the United States, the same macroeconomic and international economic trends that gave rise to the establishment of ATP also led to a rethinking by state governments of how best to promote state economic development. Many states began taking more active steps to revitalize their economies. ⁹² This led to a modest shift in emphasis from traditional state recruitment strategies that centered on bidding for existing firms through monetary incentives, such as reduced taxes, to one on generating new firms and industries. ⁹¹

Beginning in the early 1980s, a growing number of states established state technology programs. Varying somewhat in emphasis across the basic research, applied research, and commercialization continuum, these programs included a diverse set of specific programs: incubators, venture capital funds, manufacturing modernization programs, cooperative university-industry R&D programs catalyzed and subsidized by state funds, and others. ⁹³

The parallel development of state and federal government technology programs throughout the 1980s created new opportunities as well as complexities in coordination. Several of the major federal initiatives, such as the National Institute of Standards and Technology’s Manufacturing Extension Partnership program and the National Science Foundation’s Engineering Research Centers program, required state contributions, but ATP did not. Consequently, few direct linkages existed at the outset between ATP and the state programs.

To examine the relevance of state technology development programs to its operations, ATP commissioned a two-volume study of state experiences, under the title, Reinforcing Interactions Between the Advanced Technology Program and State Technology Programs, Volume 1: A Guide to State Business Assistance Programs for New Technology Creation and Commercialization, is authored by Marsha Schachtel, a Senior Fellow at the Johns Hopkins Institute for Policy Studies, and Maryann Feldman, then Associate Professor of Economics at Johns Hopkin