NIST GCR 02-834
Benefits and Costs of ATP Investments in Component-Based Software

ATP has pursued a focused program in component-based software.  The focused program is unique in that ATP is investing in a broad range of projects with a common objective.  Each project is pursuing different innovations in component-based software, but all have the same intent of bringing new technology to the software production process.  Like all risky, high-tech research, the reality is that some of these projects will produce significant new innovations and others will fail. 

The ATP program in component-based software was a 5-year effort to establish the technical foundations for fundamental change in the software industry.  Drawing on current research in automated software design and production, the program supported projects to enable a market in broadly useful software components.  The goal of the program was to make the use of software components as ubiquitous as the use of integrated circuits that make up today's component-based computer hardware.  In the new component-based world, components will be automatically combined and configured by software composition tools to mesh into large applications without requiring the user to understand how the individual components operate.  As development becomes easier, software engineers will be able to move from the mechanics of the software development process to the development of applications using automated tools to assemble and integrate independently produced components.  ATP made a variety of investments to accelerate the development of the technology. 

Because of differences in success rates, we evaluated the entire program as a portfolio of investments.  By choosing this approach we are getting an accurate picture of the total program.  If we only chose one of the companies to examine, we might drastically under- or overestimate the benefit from ATP.  For example, if we were to examine a single project that was highly successful with an internal rate of return of over 100 percent, we might conclude that all of the funded companies were highly successful.  Alternatively, if we were to examine a project which produced a negative internal rate of return, we could assume that the entire ATP program was a failure.  A portfolio approach provides the best picture of the entire program.  Table 5-1 provides a profile of the projects' ATP funding.

Section 5.2 describes the actual returns and presents quantitative measures of the economic performance of the CBSD focused program.  Perhaps more interestingly, Section 5.3 presents the lessons learned from the ATP program and describes the non-market benefits of ATP funding not directly visible in the measures of economic performance.

Return to Table of Contents.

As described in Section 1.3, eight of the 24 funded projects were selected for quantitative economic analysis.  At first it might appear that RTI used a "cherry-picking" approach by selecting only the best projects to include in the portfolio analysis.  However, this is not the case.  We estimated the benefits from the eight projects that we believed were a priori the most successful and included costs experienced for the entire set of 24 projects.  We assumed that the remaining 16 projects generated zero economic benefit.  This is an extremely conservative approach.  By assuming that these projects had no benefits and only costs, we are intentionally biasing our results downward.  If the benefits from the few selected projects were able to overcome this severe, intentional, built-in bias, we could have strong confidence that the entire focused program was successful.

5.2.1  Overall Focused Program Performance

From each of the eight in-depth case studies, we estimated the total social benefits generated by actual and projected sales of products based on the ATP technology.  This stream of benefits was then compared to the total costs of the project to generate a portfolio net benefit estimate and a portfolio internal rate of return.  Table 5-2 shows ATP and private expenditures in the component-based software program and the stream of benefits realized by the program.  Based on the data from Table 5-2, we calculated an overall net present value, benefit-cost ratio, and an internal rate of return.  These are presented in Table 5-3.

The ATP component-based software program was successful; based on a discounted investment of $34 million in ATP funds and $55 million by the private firms, the program was able to generate estimated discounted benefits in excess of $850 million.  Even this return is an underestimate of the total benefits of these investments.  It only includes benefits from eight of the projects, limits the life span of the projects, and is based on conservative estimates of the benefits from each of the individual technologies that ATP funded.

Table 5-3.  Measures of Performance of ATP-Funded CBSD Program

5.2.2  Individual Project Performance

The performance of the portfolio of projects funded in the focused program follows expectations for risky investment projects.  Some of the projects failed and barely generated any social returns, while others were quite successful.  Three of the projects-the Commerce One JV and the Tom Sawyer and Intermetrics single-company projects-generated enough returns independently to cover the entire cost of the focused program.  The benefits generated from these three projects compared to the entire investment costs generates a benefit-cost ratio of over 10.  This result is very consistent with high-risk, high-return investment.  Given the high-risk nature of new software and Internet companies during the period that ATP invested in these companies and the time when they started to release new products, the returns generated by the ATP focused program are significant.  Table 5-4 lists the estimated project returns for the eight in-depth case studies.

5.2.3  Distribution of Benefits

As the methodology and measurement portions of this report detail, economic benefits from the technologies funded by ATP's focused program were shared between the funded companies and the customers of their component-based products.  The funded firms received the capital needed to develop products they could sell profitably in the market.  Their customers were able to purchase and use component technology to lower their costs of developing software systems and information products and services.

As a part of our in-depth financial analysis of the eight CBSD firms profiled in this section, we estimated the value of both producer and consumer (customer) surplus over the relevant time horizon.  Table 5-5 presents a breakdown of these performance measures for the eight case studies.  We also briefly investigated the profitability of these firms, taking into account estimated producer surplus and their internal investments.  We found that one firm incurred a net loss, two earned small economic profits, and five made significant profits.  We would expect that the last five firms might experience competitive entry into their markets in the near future, with resulting downward pressure on their margins and net profits.

Our evaluation of this ATP program was performed very early in the product life-cycle of many of the component-based products offered by the funded firms.  Because of this advanced timing, many of the quantities and prices were future projections made by the principals we interviewed, rather than results of actual sales data.  Reference to Table 5-2 shows that a large fraction of the total consumer and producer surplus is expected to accrue in the 2001 to 2004 time period.  For this reason, we have broken down the portfolio's estimated surpluses into those already incurred and those projected into the future; these totals appear in Table 5-5.

Return to Table of Contents.

In addition to the social benefits discussed above, the funded firms reported that ATP made several contributions to their success that are more difficult to quantify.  Standardized requirements for proposal writing, record-keeping, and progress reporting caused firms to be more thorough in their project planning and execution.  The funding environment was well suited to the unpredictable nature of R&D, as both patience and flexibility on the part of ATP supported the firms' efforts.  A "halo effect" increased credibility and lowered barriers to commercialization of the technologies developed.

This section integrates information from all eight of the companies that RTI studied.  Although each company had a unique experience in developing its technology and in determining what applications and products to pursue, all of the companies shared numerous common experiences.

Proposal Writing and Organization

Even before companies received funding from ATP or started working on developing new technologies and products, they had already benefited from ATP's program.  The application process is so rigorous and thorough that companies invested significant resources in determining where the current technological opportunities were, identifying potential solutions, and envisioning a marketing plan to fill a particular market segment.  One firm stated that, by forcing companies to write effective proposals, ATP ensured that their firm put more thought into the industry, thus improving the research done in that sector.

The companies also said that the funding process was very fair.  Some companies noted that, for most federal funding processes, the company seeking the funding needs to have a personal connection to the funding agency.  Companies have to know someone at the agency, have worked with them in the past, or otherwise have had a connection.  Respondents said that ATP made their funding decisions solely on the merits of the proposal, leading to better project selection and greater returns to the program.

ATP's Understanding of the Timing and Risks of Research

Years often elapse between the time an idea occurs and when a product can reach the marketplace.  This is especially true when an entrepreneur is pursuing a groundbreaking line of research.  However, most venture capitalists want to see a return on their investment within 2 years.  In addition, they want a high degree of certainty about the probability of success of the research that they are funding.

In this competitive world of funding R&D of new technologies, projects that are high-risk/high-return but several years out are rarely funded by venture capital.  ATP understands this problem.  ATP's funding of the risky phases of technology development, where products are many years off, gives companies a chance to pursue ideas that the private sector will not fund.  Tom Sawyer Software is an example of how ATP's patience generates returns.  ATP first started funding Tom Sawyer in 1995, and they have only recently brought a product to market.  However, the expected returns on this ATP project are so large that the benefit-cost ratio is projected to be 18.  Without ATP's patient capital, a company like Tom Sawyer might never have been able to develop the unique technology embodied in their product.

ATP is able to circumvent the problems associated with the risk and uncertainty of a product by investing in a portfolio of projects.  This investment approach allows them to select from the best projects and not worry about the timing or risk that is associated with individual projects.  ATP closely examines risk and uncertainty but, unlike a venture capitalist, does not eliminate a project just because it is risky.

Flexibility

ATP also understands the flexible nature of R&D.  A company might be pursuing a strain of research that appears to be promising.  At one point the company might realize that it needs to change the focus of the research in a new but related direction.  ATP gives companies the flexibility to change their research midstream as conditions merit, with oversight from ATP's project management team and maintaining consistency with the original project goals. 

For example, one of the firms that achieved technical success with its technology did not achieve commercial success with an early spin-out product.  ATP allowed the firm to use the technical knowledge gained from the failed spin-out product to develop new project tasks, equivalent in technical and business merit and faithful to the overall project goals.  The firm has yet to develop an effective commercial product, but it is still engaging in cutting-edge research with non-ATP funds that may reap dividends in the future.

Acceleration versus Feasibility

The companies that RTI examined all said that the ATP funding was instrumental in accomplishing their projects.  SciComp, which developed a successful product called SciFinance, strongly felt that without ATP, its technology never would have developed in any form.  Venture capitalists were unwilling to wait the seven years required for development, and no other institutions were willing to invest in SciComp.  This firm now expects to generate revenues of over $10 million per year in the near future.

Other companies said that without the ATP funding they probably would have developed very different products, if any at all.  For example, without the ATP funding one existing firm would have continued to sell a small piece of conventional software used by a handful of programmers.  With the funding, this company has used component technology to develop a sophisticated and elaborate product that is used to control the pre-launch sequence for the space shuttle.  Enabling technology that makes new product ideas feasible and possible, rather than simply accelerating the adoption date of one product, greatly increases the benefits attributable to ATP.

Halo Effect

The research community often views ATP as being an impartial judge of quality.  When ATP funds a particular company it is vouching for the quality of the company's research ideas and their business and economic potential.  When potential customers are making purchase decisions they may consider this information.  This "halo effect" should translate to more sales and more opportunities for the funded companies.

Several companies said that the ATP funding generated a halo effect.  The halo effect can emerge in different ways.  One firm, which was a start-up company when it received the ATP grant, said that the ATP grant gave them credibility with venture capitalists and other financing mechanisms.  A second start-up found out that ATP gave them enough credibility with their customers that they were able to charge a price premium for their higher-quality products.  An established firm noted that the ATP funding gave them internal credibility with which they were able to generate more funds to conduct their R&D and expand the scope of the project.

Return to Table of Contents.

5.4   CONCLUSION

The quantitative and qualitative analyses presented in this report demonstrate the significant impact of ATP's investment in component-based software technology.  A large fraction (two-thirds) of the funded projects achieved their technical objectives; many of these firms and one notable JV proceeded to develop successful commercial products based on the technologies developed.  ATP was credited by the firms with enabling the R&D efforts of many of these firms, accelerating the technology development process, and increasing the probability of technical and commercial success.

When the 24 funded projects are viewed as an investment portfolio, the social returns exceed any reasonable benchmarks for public or private investment.  The estimated 80 percent internal rate of return reflects a substantial benefit to the nation in excess of the return to the companies funded and is an indication of inefficient capital markets for projects with high technical risks.  The calculated net present value (NPV) of $840 million and benefit to cost ratio (B/C) of 10.5 suggest that the funds were a worthwhile expenditure of public funds.

Return to Table of Contents or go to next Chapter.

Date created: December 3, 2002
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)