NIST GCR 02-830
Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms
A Framework Using Both U.S. and Japanese Data

• Stimulate_creative_thinking—To what extent has collaboration enabled your firm to stimulate creative thinking? Here, and in most cases below, firms were asked to give an ordinal response, i.e., “significantly,” “moderately,” “little/none,” or “unsure.”
  
• Avoid_redundant_R&D—To what extent has collaboration enabled your firm to avoid redundant R&D expenses?
  
• R&D_cost_saving—How much (in dollars) has your company saved in R&D expenses through collaboration?
  
• Time_saving—To what extent has collaboration allowed your company to save time in general?
  
• Delayed_product_entry—To what extent has collaboration delayed product entry into the marketplace?
  
• Delayed_R&D_phase—To what extent has collaboration contributed to a delay in the R&D phase?
  

To protect the confidentiality of firms’ responses, ATP disguised firm identity in these data. As a result, we averaged firm-level responses by consortium and linked this averaged response to other data.

The firms’ responses came from different ATP reports; some came from the closeout report, others came from reports during the operation of the project. The relative success or failure of a consortium may affect a manager’s perceptions of the consortium’s attributes; thus, these qualitative data are not necessarily exogenous, in a statistical sense, to our measures of project outcomes. Nevertheless, we present our regressions as a way of assessing statistical relationships between these variables and research outcomes, without necessarily being able to prove anything about the causal nature of the relationship.

MODEL

The basic empirical model used in this section is as follows:

(3)

Here P_it denotes the sum of U.S. patent grants generated by member firms of consortium i in year t in the technological areas targeted by the consortium. P_it is given as a simple count. Budget_it represents the total research resources expended by the consortium. Pre_project_patenting_i denotes the average patenting in the targeted classes, with the average taken over a 5-year window prior to the official start date of the project. Our qualitative variables are represented in the sigma term.

The spillover-inducing effect of the consortia is captured with a method borrowed from the macroeconomics literature on impulse response function.⁶ A skeptical view of the benefits of research consortia maintains that any positive impact on the innovative output of participating firms is produced entirely by the combination of subsidies granted to the participants and the research resources expended by the participating firms out of their own R&D budgets. A positive view of the benefits of research consortia maintains that, in addition to the public and private financial resources expended, the process of bringing firms with complementary research assets into contact should itself enhance the innovative activity of the firms involved.

Project_duration_it is a dummy variable set equal to one during the years, t, for which project i is active. The regression coefficient on project_duration_it measures the boost to patenting in the targeted area sustained by the participating firms during the duration of the project. If this variable is positive and significant, then we interpret this as evidence that participation in consortia promotes spillovers of complementary knowledge among participants, enhancing the productivity of their collective research effort, and that these effects generate a more or less immediate impact on the patenting of consortium members.

Presumably, a boost to patenting in the targeted area that endures past the conclusion of the project is of greater social value than one that ends with the project. To get a sense of the average time path of benefits, we constructed two additional variables: years_passedit and years_passed2_it. Years_passedit measures the years elapsed since the start of a given project. A positive coefficient on years_passed_it and a negative coefficient on years_passed2it imply that the boost to patenting has a quadratic shape: it rises initially after the start of the project, but peaks and then declines at some later date.⁷ A large, positive coefficient on years_passed_it and a relatively small, negative coefficient on years_passed2_it would imply that a relatively long-lasting boost is obtained from consortia participation.

Another way to estimate the time path of benefits is to include a number of dummy variables corresponding to periods of set length after the inception of a project, and to examine the coefficients on these various dummies. This more flexible approach does not impose a quadratic structure on the benefits stream. Due to the time-truncation problem in our U.S. data, we are only able to estimate lags up to four years in length.

RESULTS

In our initial regression, we estimate the “stripped down” version of equation (3), leaving out, for the moment, our qualitative data on the nature of the consortia. Results are given in column 1 of Table 3. A Poisson regression model is used to explicitly allow for the “count data” characteristics of the dependent variable. The coefficient on project_duration has the interpretation of the additional patents generated per year that a consortium is in operation. In column 1, the coefficient on project_duration is positive and significant, though rather small in magnitude.⁸

The coefficient on budget is negative and significant, suggesting that a larger budget for a consortium paradoxically results in fewer patents. We believe this to be an artifact of the data, driven by trends in ATP’s budget. When ATP began sponsoring its first projects during the first Bush Administration, it had a modest budget. The ATP considerably expanded during the Clinton Administration; however, our patent coverage for these later years is truncated. The negative coefficient for budget, which appears in most of our succeeding specifications, is probably driven by truncation of patents resulting from later projects rather than strongly decreasing returns to R&D subsidies.

The coefficient on years_passed is positive and significant, and the coefficient on years_passed² is negative and significant. These coefficients imply a boost to patenting that is small initially, grows fairly rapidly, peaks within 2–3 years of the inception of a project, and declines thereafter. Our interpretation of the coefficients on project_duration, years_passed, and years_passed² are all clouded by the time truncation problem in our patent data. With a longer post-project time series, we may very well observe more long-lived effects.

In column 2, we implement a more flexible version of the impulse response function, using dummy variables to represent time lags instead of the quadratic form used in column 1. The results of this model are quite consistent with the results of the quadratic time path of benefits estimated. The impact of participation in the first year is small and not statistically significant. Thereafter, the measured impact grows rapidly, peaking in the second year, and declining rapidly thereafter. Because of the time truncation in our data, the actual decline may be less rapid than the apparent decline shown in this model. This concern notwithstanding, these results support the evidence presented in the previous section: participation in ATP-funded consortia has a positive, statistically significant impact on patenting by participating firms in the targeted technology.

Column 3 adds our measure of average technological proximity to our baseline specification. Column 4 adds calendar year as a control variable to capture any overall time trend in patenting. In both cases, higher measured technological proximity is positively and significantly associated with higher levels of patenting in the targeted technologies. This suggests that the measure of average proximity we constructed here can be used to help predict the likelihood of project success ex ante.

We test the robustness of our baseline specification by including indirect resources received by project participants from other overlapping projects. Results are given in column 5. The basic results of our regression are unaffected by including this additional control variable.

Lastly, we include firms’ responses to survey questions in the BRS while controlling for project budget and pre_project_patenting in the targeted area. Results presented in Table 4 are grouped by survey content. The model in column 1 shows that the coefficient on stimulate_creative_thinking is positively and significantly associated with patenting in the targeted area. The model in column 2 includes the variables avoid_redundant_R&D, R&D_cost_saving, and time_saving. Two of the three variables are positively and significantly correlated with project outcomes, suggesting that the perception of enhanced efficiency from collaboration is indeed correlated with higher levels of patenting in the targeted areas. The coefficient on avoid_redundant_R&D is negative. This result is difficult to interpret without further research, and is potentially contaminated by its collinearity with R&D_cost_saving. The model in column 3 measures the association between perceptions of problems or delays due to collaboration and project outcomes. As expected, the coefficient on delayed_product_entry is negative, suggesting that perceptions of delays in getting products to the marketplace as a result of collaboration are associated with reduced patenting outcomes. In contrast, the coefficient on delayed_R&D_phase is positive, suggesting that perceptions of delays in the R&D phase of a project are associated with increased patenting outcomes. Although this result is difficult to explain without additional research, it is possible that technologically ambitious projects tend to fall behind schedule, but in the end generate greater technological payoffs.⁹

IMPLICATIONS

In this section, we showed that the establishment of an ATP-funded research consortium stimulates patenting by participating firms in the targeted technological areas. We identified some characteristics of consortia that lead to greater relative levels of success. Our results suggest that technological proximity, as measured by the degree to which the patenting portfolios of participating firms are similar, fosters spillovers; this, in turn, enhances the research productivity of the participating firms in the consortium. We also find a strong link between pre-consortium patenting in the targeted area and subsequent success. These results suggest that policy makers can use technological proximity and pre-consortium patenting in the targeted area as a criterion for selecting projects and member firms.¹⁰

We established a statistical link between some of the survey response variables in the BRS and quantitative outcome measures. The relationship between total project budgets and total project outcomes is less clear. The ATP faces a tradeoff between investing relatively large amounts of money in a small number of projects versus investing smaller amounts of money in larger numbers of projects. The evidence we present does not suggest that the projects with the biggest budgets generate the highest levels of patent output. However, the measured relationship between budget and outcomes is quite possibly distorted in our data by the time truncation problem in our patent data. More research will be necessary to clarify the nature of this important relationship.

NOTES:

1. An alternative to this measure would be revenues obtained by corporations from the sales of products whose design and development were stimulated by participation in consortia. The mapping from consortia to commercial products, however, is a challenging task because even the participating firms themselves would have difficulty tracing the changes in their annual revenues due to products growing out of their participation in a consortium.
2. We acknowledge with gratitude the superb work done for us on this by the staff of Bailey Services, Inc.
3. However, it is certainly possible that there may be important technological complementarities between “distant” technologies that this index fails to measure.
4. We do not (and cannot) control for the spillover-stimulating effect of overlapping consortia. Thus, the estimated “spillover” effect attributed to one project may, in fact, partially reflect the “spillover enhancing” impact of the overlapping consortia.
5. In early 1994, ATP implemented the Business Reporting System (BRS), an electronically administered data collection tool for tracking progress of projects selected for ATP awards from 1993 to the present. BRS tracks the progress participants are making on their business plans and projected economic benefits that were originally outlined in their project proposals and updated over the course of conducting the research. Data are collected on a routine and regular basis at the individual participant level within a project to ensure maximum confidentiality of information.
6. We thank Oscar Jorda of UC-Davis for this suggestion.
7. It may be that the impact on firm patenting observed during the duration of the project is negligible, such that the estimated coefficient on project_duration is small. However, after the official cessation of the project, we may observe a substantial increase in patenting in the targeted area, as the research results obtained through the consortia are incorporated in the firms’ own research programs.
8. Recall in the Poisson model, the regression coefficients have a semi-elasticity interpretation. The coefficients represent the percentage change in firm patenting associated with a unit change in the independent variable.
   
9. To facilitate interpretation of the coefficients on the survey response variables, we restrict our time dimension in these regressions to observations after the start of an individual project.
10. In other regressions not shown here, we estimated the elasticity of consortium outcomes with respect to pre-consortium patenting to be approximately 100%.

Date created: January 24, 2003
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)