NISTIR 6917
Different Timelines for Different Technologies:
Evidence from the Advanced Technology Program

APPENDIX. Actual Versus Expected Timelines: Preliminary Results

Expectations alone are an incomplete indicator of the timeline for commercialization—particularly given the risky R&D projects ATP funds. Early on there are considerable uncertainties about achieving the technical goals. Technical hurdles, and additional capital requirements, may delay the projects beyond the window of opportunity. ATP does not seek to fund projects with high business risks, but some risks are unavoidable.

ATP’s objective is to fund breakthrough, enabling technologies, some of which will serve as platforms for emerging industries. Many projects target markets that are just emerging or are almost non-existent. Business risk is an inherent characteristic of such projects. Moreover, technologies may be already ahead of the market or ahead of critical complementary technologies and products. Unexpected competition may arise.

Enough ATP projects have now been completed to begin comparing actual commercialization activity with expectations. Here, we take a small step forward. This Appendix provides a preliminary assessment of the question: To what extent is actual commercialization mapping to plans? Data are still very limited. This is the first study to use BRS data to track a given group of projects over time, from the beginning of the ATP funding into the post-ATP funding period. Some initial observations are possible. Given its preliminary nature, this analysis seems most suited to an Appendix. More in-depth analysis will follow as more data become available.

We identified Close-out Reports to match Baseline Reports for approximately one-third of all the applications analyzed in the body of the study. (See discussion of these reports in Section II A.) We used these paired reports to compare actual commercialization experience (as of the end of ATP funding) with expectations at the start of the projects.

Similarly, we had Post-project Surveys to match approximately one-third of the project participants whose reports are analyzed in the body of the study. The Post-project Surveys were conducted two to three years after the end of ATP funding. They enabled comparison of actual commercialization experience as of two to three years after the end of ATP funding with expectations in the ATP-funding period.

A. Very Early Commercialization—During ATP

Our reduced data set of matched Close-out and Baseline Report data covered 301 commercial applications reported by 176 organizations in 147 projects. We compared the characteristics of the matched Close-out and Baseline Reports and compared actual business progress reported in the Close-out Reports with expectations about the timing of commercialization reported in the Baseline Reports.

Profile of Matched Commercial Applications

The distributions by size of firm, technology, project structure, and type of application are shown in Table A.1.

Table A.1. Profile of Commercial Applications in Matched Closeout and Baseline Reports

The distributions for this reduced, matched set of Baseline and Close-out reports can be summarized and compared with the full set, as follows:

• These 301 applications are about equally divided between small and medium-large-sized firms. Just over half the applications were reported by small companies, approximately the same percentage as for the full group analyzed in Section IV.
  
• Nearly one-third are applications of materials-chemistry technologies, while a quarter of the applications involve information technologies. Proportions of biotech and materials-chemistry applications were somewhat higher, and proportions of electronics/photonics and manufacturing were somewhat lower than for the larger group of applications analyzed in Section IV.
  
• Over two-thirds (68%) of these applications come from single-company projects. Applications from participants in joint-venture projects are poorly represented given that applications from single-company projects represent only 39% of the 1,172 applications analyzed in Section IV, but 68% of the smaller, matched group of Close-out and Baseline reports.
  

Baseline Expectations versus Close-out Performance
For evidence of commercialization as of the end of the ATP project, we used responses to the following questions in the Close-out Reports:

• Have you earned revenues from products/services to date? (Y/N?)
  
• Have you to date begun production (of a new product or in a new process)? (Y/N?)
  

The results are shown in Table A.2 and Figure A.1. These results show that at the beginning of ATP funding, revenues were expected for only 14% of the 301 matched applications during the ATP-funding period, ranging from a high of 29% for IT technologies to a low of 5% for biotechnology applications. By the project Close-outs, 22% of the matched set of applications had generated some revenue from their initial, first-generation applications.

Figure A.1. Comparison of Actual Revenue Activity at Project Closeout with Expectations at Baseline

All technology areas except manufacturing had earned revenue for more applications than expected at this early stage. IT and electronics-based applications showed the most commercial activity, with 33% of IT-based applications and 31% of electronics-photonics-based applications reporting some revenue.

A few caveats are in order. It is evident from the information provided in Closeout Reports about actual amounts of revenue to date that much of the activity was at a low level.

The applications for which revenue was reported represented the earliest opportunities to commercialize early stages of the technology. Almost no company reported more than one application with revenues (data not shown). On the other hand, the results may undercount actual commercial activity because the revenue question is not directly applicable to (or posed for) process applications that may have been implemented.

Early revenues sometimes derive from sales of samples or relatively early prototypes, rather than actual commercial production. To compensate for limitations in the earned revenue variable, we examined responses to the question “Has production begun?” This question applies to process as well as product applications, but not to service applications. In general, the production results shown in Table A.2 are consistent with the revenue earned results, but show somewhat lower levels of activity.

Across all applications, actual production appeared to be occurring ahead of baseline expectations of revenue. Production activity was reported for 17% of applications, compared with baseline expectations of 14%.

Biotechnologies showed much more early production activity than any other technology area. This early biotech activity was likely associated with the fluid stages of innovation, highly customized products, and new markets being served. Perhaps the vigorous venture capital activity in the late 1990s facilitated this early activity.

Production results for manufacturing technologies and information technologies showed somewhat fewer applications in actual production than baseline expectations suggested.

The omission of service applications in this variable may account for the IT result. The manufacturing applications that had earned revenues as of the end of ATP funding were likely only in pilot production.

Biotechnology companies were in production for some applications that had not yet actually generated revenues. One would expect this of process applications, but not of the product applications that are more typical of biotech firms. More investigation is needed, but the early production using biotechnologies may simply reflect different definitions biotech and manufacturing-based companies apply to “production” in the early stages of their product cycles.

The key result is that when these matched applications are considered as a whole, at least as many applications as was expected resulted in actual commercial activity at ATP-project closeout—for each technology area. The many small companies were seeking to raise market awareness for very early generations of their products and to increase cash flow. There is considerable competitive pressure to be first to market in new or emerging markets. Larger-volume, higher-value applications would come later.

B. Longer-Term Commercialization Activity—Two to Three Years after ATP

Post-project Surveys generate considerable detail about commercial activity after ATP funding ends. Among the 558 ATP-project participants covered in the body of the study, we identified 173 companies (from 135 projects) that were covered in the Post-project Surveys completed as of early 2002. These surveys covered participants in ATP projects that were completed by the end of 1998 and thus were eligible for their first Post-project Survey by 2001. Survey results were available from approximately 95% of single and joint-venture project participants in the eligible group. We used the Post-project Survey data to perform a longer-term comparison of actual commercialization with plans.

Profile of Matched Project Participants

Table A.3 compares the distribution of participants in these Post-project Survey data with the 558 participants whose commercialization plans are analyzed in the body of the study.

The similarities and differences in portfolio characteristics in the total set of 558 participants and the reduced set of 173 post-project participants can be summarized as follows:

• The entire group of 558 and the post-project group were both about equally divided between small and medium-sized firms. A very small number of non-profit organizations were included in each.
  
• Two-thirds of the post-project group were single-company awardees, but only one third of the larger group were single-company awardees. The Post-project Surveys reflected a much larger percentage of single-company projects because those projects and participants were a larger fraction of the first completed ATP projects and thus the first to become eligible for Post-project Surveys. The average number of participants per project was considerably smaller for the post-project group because there were relatively few joint ventures.
  
• Percentage distributions by technology area were quite similar for the larger group and the post-project group. The major differences were that there was a smaller percentage of manufacturing participants in the post-project group and a larger percentage of materials-chemistry participants. This difference disappeared in looking at the project distribution by technology area. It likely had little impact on the comparability of the post-project and earlier data.

C. Post-project Performance

For evidence of commercialization in the post-ATP period, we used responses to these questions in the Post-project Survey:

• Has your organization earned any revenue (or experienced reduced costs of production of goods for sale) to date as a result of its ATP project? (Y/N)
  

For those who said NO:

• In the next two years, does your organization anticipate earning revenue from its ATP-funded technology? (Yes, No, Don’t Know)

For those who said NO:

• In the next 2 to 10 years, does your organization plan to introduce product/license, or introduce a process in house that incorporates ATP-funded technology? (Yes, No, Don’t Know)
  

The following is a summary of the responses from the 173 organizations:

• 46% reported they had received revenues (or cost reductions from new processes) from 104 applications in their first Post-project Survey.
  
• 14% expected to launch products in the next two years, and 12% planned to introduce a product in the subsequent 2 to 10 years.
  
• 28% no longer anticipated revenues from their ATP-funded technology or were very uncertain about it.
  
• 78% of the applications that have earned revenues to date involve new products or services; the rest are about evenly split between manufacturing processes and patent licensing.

Table A.4 provides a detailed breakdown by technology area. Given the small number of participants in each area, the results must be considered preliminary. Nevertheless, it appears that the major portion of the companies reporting who will commercialize their ATP-funded technologies have done so, at least for their first applications.

This preliminary analysis strongly suggests that a large portion of the commercialization activity in all technology areas occurs, or at least begins, by two to three years after ATP funding ends. Nearly nine out of ten Information Technology participants who still expected to commercialize their ATP-funded technology had already done it. Two out of three electronics/photonics participants who still expected to commercialize had done it. In both areas, no further new commercialization was expected after the next two years. Biotechnology and materials-chemistry participants, on the other hand, expected considerable new activity in the next two to ten years. Over one in three electronics/photonics and manufacturing participants no longer expected any revenues at all or were very uncertain as to their timing.

In general, the pattern is consistent with the results presented in the body of the study. Information technologies and electronics/photonics technologies either commercialized quickly or not at all. Biotechnologies had considerable early commercialization activity for at least some types of applications, but many companies saw a long window of opportunity and commercialization timeline. The number of data points was small for manufacturing participants; however, it appears that there were a large number who had cancelled plans to commercialize. Additional investigation is needed to determine the reasons, but it may be that the cost targets were not met or had shifted so that the technologies developed with ATP funding were not deemed commercially viable.

Note that this analysis was conducted by summarizing activity by project participant rather than by individual commercial applications, as was done in the Baseline-Close-out comparisons. By two to three years after ATP funding ends, markets have changed substantially since Baseline reports were submitted. Therefore, the Post-project Surveys do not specifically relate applications reported in the Post-project Survey with those reported at the start of the ATP project. Nevertheless, the distribution of applications by type (product, service, or process) was similar for the Post-project, Close-out, and larger body of data. Considerable application-level information is available in the Post-project Surveys and should become part of future analyses.

Finally, we considered whether the ATP project participants were satisfied with their progress towards commercialization—that is, to what extent they felt progress had met their expectations at that point in time. We examined responses to the following question:

• Achieved beyond your goals?
  
• Achieved your goals?
  
• Achieved less than your goals?
  
• Don’t know.
  

Results can be summarized as follows:

• 11% had exceeded their business goals.
  
• 24% had achieved their business goals.
  
• 65% had failed to achieve their goals.
  

Reasons for failure to acheive goals were quite evenly spread across a variety of business and technical factors.

Although the Close-out Reports had indicated that the timing of initial revenue activity had exceeded expectations, the pace seemed to slow down after ATP funding ended. Cumulative commercial activity for most project participants was somewhat less than expected by two to three years after ATP funding ended. Looking at the data from the ATP-funding period (Section IV) together with the Post-project data, 66% of companies expected revenues from at least one application within two years after ATP; however, only about 54% of the companies in the Post-project group had achieved commercialization by two to three years after ATP. Of the remainder, half still expected to earn revenue (most in the next two years); the other half apparently had largely given up hope. Sixty-five percent reported they had failed to achieve their goals for this point in time.

Are expectations about commercialization patterns of ATP projects being met? Significantly more data are needed, both aggregate and by technology area, to answer this question. For example, the Post-project data available reflected a much larger percentage of smaller, single-company projects than the data set used in the body of the study.

Furthermore, the data presented in this study do not address the magnitude of commercialization efforts or actual benefits to the ATP-funded organizations or the nation.

Given the risky nature of ATP projects, and the relatively long timelines to commercialization, not all projects will be successful in accomplishing the technical goals needed to achieve maximum economic impact. For many, substantial technical barriers remain after ATP funding ends. Those that do meet their technical goals will face business hazards during the long planning and development period—hazards that may force them to miss the market window of opportunity.

The early data suggests that a large proportion of the projects that will ultimately achieve commercialization will do so, for their initial applications, within two to three years after ATP funding ends. The momentum of some project participants will continue along longer time paths. Two to three years after ATP funding ends, a significant number will have dropped plans for their ATP-funded technologies or are highly uncertain about them.

Overall, the emerging patterns of actual commercialization are consistent with expectations about timing of commercialization presented in the body of the study. Further studies will examine whether the different trajectories for different technologies, and associated commercialization strategies and objectives, remain consistent with the innovation life-cycle model over the longer term.

Return to Table of Contents or go to References.

Date created: March 4, 2003
Last updated: April 12, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)