A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 7:  Econometric/Statistical Method

Econometric/statistical analysis methods allow researchers to test the strength of economic relationships and to understand the range of variability in the estimates. As explained in Chapter 2, the strength of these methods is that they can provide more statistically defensible evidence about expected cause-effect relationships than most other evaluation methods. For this reason, econometric/statistical analysis methods hold particular value for ATP, which seeks to determine causeeffect relationships that underlie its program, and to answer pressing stakeholder questions about the program’s effects on firms, industries, technological innovation, the U.S. innovation system, and the national economy.

In practice, these methods are data-intensive. Considerable effort is often required to obtain, array, and adjust the data that will be used to test the hypothesized relationships. The tasks involved in data collection and adjustment are not only expensive and time consuming, but can also involve numerous decisions and assumptions that significantly affect findings. Often it can be difficult to make the resulting analysis sufficiently transparent to those whose actions may be guided by the findings. A noteworthy aspect of several studies described here is the authors’ candor in acknowledging that data were not obtainable to test certain models, that findings are tentative, that sample sizes were small, or that the time period studied was short. These self-assessments are reflective of the exploratory nature of ATP’s use of the econometric/statistical analysis method.

The chapter is organized around the key questions that the studies were intended to answer. Table 7–1 lists eight studies commissioned by ATP that in this chapter illustrate the use of econometric/statistical methods in evaluation. The table provides a quick reference to study purpose and techniques used. While the presented studies illustrate specific econometric techniques, they in no way provide comprehensive coverage of this broad topic.

Table 7–1. Eight of Ten Studies Using Econometric/Statistical Methods Represented*

Note: Studies by Austin and Macauley, 2000; and Fogarty, et al., 2000 draft, also used econometric/statistical methods, but they are treated under “Emerging Methods” in Chapter 8.

Testing ATP’s Leveraging Effects on Advanced Technology Development

The Feldman-Kelley study ¹⁸³ was introduced in Chapter 5 as an example of the survey method, but it is also included in this chapter since it illustrates use of econometric/statistical analysis to test the strength of hypothesized relationships. The study’s tested hypotheses center on whether there is a difference in project characteristics and firm practices between firms that receive ATP awards and those that do not, and whether ATP funding makes a difference to firms in attracting additional resources. Feldman and Kelley employed multivariate analysis with control variables to provide a more stringent test of the validity of the hypothesized relationships suggested by survey findings.

Testing Whether ATP Winners and Non-Winners Differ in their Project Characteristics and Practices

Feldman and Kelley hypothesized that the following four attributes of an R&D strategy are indicative of an approach conducive to achieving ATP objectives: (1) participation in inter-organizational networks, (2) willingness to share information and to transfer it to other firms, (3) establishment of new collaborative partnerships in the project proposed, and (4) proposal of a project that is a departure from the rest of the firm’s research portfolio. They wanted to know if a higher incidence of these attributes would be found in winning firms and projects than in those applicants that did not succeed in getting an award. The survey asked questions to determine the strength of inter-organizational linkages, the tendency toward openness or secrecy, the creation of new partnerships, and the submission of proposals in new technical areas. Survey results were tabulated and tests of significance performed. Analysis of descriptive statistics showed that ATP award winners were more likely to have the four attributes than non-winners, with high statistical levels of significance.

The researchers then carried out multivariate regression analysis, using logistic regression, to test the strength of the four attributes they had identified as factors influencing a firm’s chances of winning an ATP award. They ran the model for three cases: (1) a base case with no controls for other factors, (2) a case controlling for past experience of applicants with ATP and with the technology area proposed, and (3) a case that adds to the controls of the second case the additional control of ATP reviewer assessment scores as proxies for the overall quality of the proposal and firm.

The multivariate regression analysis produced robust support for the hypothesis that award winners tend to be especially strong in the four attributes, and, hence, particularly well positioned to deliver public benefits from research projects that are more likely to involve new research areas and new partners than non-winners. Other findings of the regression analysis provided evidence that a firm’s chance of winning an ATP award is not significantly improved by: (1) having applied to ATP in the past, (2) having been successful in past applications, or (3) spending more than non-winners on proposal preparation.

Testing Whether ATP Funding Makes a Difference

Concluding that winners differed significantly from non-winners in characteristics desirable for ATP, the researchers next presented survey findings dealing with the difference made by ATP funding. They focused on two questions: (1) how often do non-winners proceed with the proposed project as planned, and (2) compared to award winners, how successful are non-winners in attracting other sources of funding for the projects that were proposed to ATP? The tabulated survey results revealed that most non-winners did not proceed with any aspect of the proposed project or did so on a smaller scale. The survey results also indicated that fewer award winners than non-winners pursued other funding sources for their projects in the year following the ATP competition, but those who did were more than twice as likely as non-winners to attract funding.

Additional survey questions concerned winners’ and non-winners’ perception of the fairness of the selection process (most in both categories thought ATP’s selection process was fair), their future plans to apply to ATP (the majority said they “definitely or very likely” would reapply), and whether non-winners participating in a debriefing with ATP found it helpful (most found it “very helpful” or “reasonably helpful”).

From the survey findings, Feldman and Kelley hypothesized that the ATP award conveys a “halo effect” to award winners that helps them attract additional funds to their R&D project. But a halo effect is not directly observable, and the researchers used econometric/statistical analysis to test its existence. They specified a multivariate regression model that controls for other factors that may influence the firm’s effectiveness in attracting additional funding from external sources. These factors included a history of success in R&D fundraising; small firm size that is associated with a dependency on external funding sources together with the availability of funding sources targeting small firms; and organizational stability that decreases business risk and thereby increases the willingness of external sources to provide funding.

The researchers estimated three regression models to investigate whether or not a halo effect exists and, if so, the strength of the effect. Model 1 controlled for: (1) whether the firm is a small business, eligible for funding from the Small Business Innovation Research and other programs targeting small entrepreneurial firms; (2) the age of the firm as a proxy for the risk of business failure; (3) the amount of external funding the firm received from non-ATP sources two years prior to ATP application as an indicator of the success of its fundraising history; and (4) the maximum scores given the proposal by ATP reviewers, as a proxy for quality differences. Model 2 included the four controls of model 1, plus a variable that distinguishes ATP award winners from their non-winning counterparts. Model 3 built on model 2 to add a set of variables reflecting major technical areas proposed to control for the popularity of technology, and, hence, their possibly greater appeal to potential funding sources. The researchers used the Tobit technique ¹⁸⁴ to provide unbiased estimates of the relationships.

Feldman and Kelley concluded from the results of the three regression models that small firms attract more funding from non-ATP sources than other firms. They concluded that the firms’ track record in fundraising is positively related to their ability to attract more funds. They found that the age of the firm matters as a predictor of a firm’s ability to raise funds only when they also control for the technology area. They found that the ATP reviewer scores do not explain future fundraising success of the firms from non-ATP sources. Controlling for these other factors, they confirmed their hypothesis that winning an ATP award conveys a halo effect, allowing winners to further leverage ATP funding. In their words: ¹⁸⁵

All else being equal, ATP award winners are more successful in raising funds for their projects from non-ATP sources than firms in our comparison group. These results support our thesis that the NIST/ATP selection process produces valuable information about R&D project quality and provides an information signal that other agents find credible and are willing to act upon. Furthermore, the ATP selection signal has information content beyond that provided by technical and business reviewer ratings. Through their investment decisions, other funders, private and public, are showing, by their actions, that they believe the ATP award provides additional information about the quality of the project. (p. 39)

Modeling Impacts of Public-Private Partnerships on Firm Productivity

It was noted in Chapter 4, in the initial discussion of the Griliches-Regev-Trajtenberg study ¹⁸⁶ of the impact of government supported R&D on output, that prior studies had produced somewhat negative or contradictory findings. The researchers attempted a fresh approach to this question that is important to understanding ATP’s overall effect. Because of ATP data limitations, they demonstrated their approach using comparative data from Israel.

Griliches-Regev-Trajtenberg Model

Griliches, Regev, and Trajtenberg identified two research perspectives for approaching the question of what difference is made by government R&D funding: (1) “looking at the firms’ own R&D expenditures and asking what happened to them as a result of the availability of governmental support” and (2) looking “for differential productivity effects between own and government-supported R&D.” ¹⁸⁷ The researchers noted that the first perspective “assumes that only total R&D matters and that privately financed and governmentsupported R&D are perfect substitutes.” ¹⁸⁸ They noted that their inquiry along these lines “found that one dollar of government subsidy for R&D expands the firm’s own R&D by $0.83 and that the difference relative to 1 is not statistically significant. ¹⁸⁹ In contrast, the second perspective “denies that the source of funding does not matter and looks for differences in the effectiveness with which such funds are used by firms.” ¹⁹⁰ As the authors noted:

Governmentally supported R&D may be used less efficiently if it is subject to various constraints or if entrepreneurs do not treat grants as ‘their’ money. It could, on the other hand, yield a higher rate of return if both the application and the selection processes choose the more promising projects, i.e., if the agencies can actually “pick winners. “ (p. 8)

Applying the second research perspective, Griliches, Regev, and Trajtenberg employed the concept of effective R&D capital, where certain R&D expenditures may create more or less capital than is indicated by the amount of funding. The effect of government support of R&D on a firm’s output is thus seen as a function of the extent to which this R&D enhances output (i.e., is effective). They specified effective R&D capital as follows:

Re = Ro + (1 + d )Rg = RT (1 + d s) (1)

where Ro and Rg are own and government-granted R&D capital, respectively; d is the effective premium or discount on supported R&D; Ro + Rg = RT is the total reported R&D; and s = Rg/RT is the share of R&D grants in total R&D expenditures. If effective R&D enters the production function logarithmically, then they rewrite its logarithm approximately as: log Re = log RT (1 + d s) = ~ log RT + d s, provided the last term is sufficiently small. ¹⁹¹

They expressed a Cobb-Douglas production function, in which R&D capital services are entered as an input, as follows:

log y = · c k b k + g log RT + g log (1 + d s) (2)

where g expresses the effectiveness (elasticity) of total R&D. ¹⁹²

Demonstrating the Model with Israeli Data

Data to estimate this equation were drawn from what is described as a unique panel set of firm-level data generated by surveys performed by Israel’s Central Bureau of Statistics and the Ministry of Industry and Trade. The dataset “brings together statistics from various sources.” ¹⁹³ The use of a panel was seen as “especially appropriate in a study on the implications of R&D support for a firm’s performance, because it reveals correspondences between productivity at different points in time and R&D investments, activity, and funding type in previous periods.” ¹⁹⁴ The use of a panel also permitted construction of a set of comparison groups, including those firms that conducted R&D and which reported receiving government grants, firms that conducted R&D but which did not receive grants, and firms that did not report any formal R&D activity. The study’s findings, based on data from Israeli programs similar to ATP, suggested that the impact on firm productivity of government support to private firm R&D is positive.

Spillovers, Appropriability, and Firm Productivity

Spillovers play a central role in justifying public support for R&D, yet they are difficult to identify and measure. Improving the methods of quantifying spillovers is an important goal for a public R&D program like ATP.

In their study, Wesley Cohen, Carnegie Mellon University, and John Walsh, University of Illinois-Chicago, ¹⁹⁵ linked consideration of spillovers to that of appropriability. That is, they linked the degree to which firms are able to protect the profitability of their own inventions and the strategies they use to achieve appropriability.

Cohen and Walsh attempted to control for the negative relationships between R&D appropriability and R&D knowledge flows, thus isolating the knowledge spillover effects of R&D on the productivity of R&D at the industry level. Their objective was germane to ATP’s core mission of increasing the returns to industrylevel R&D and technological innovation (as opposed to producing benefits appropriable only by a single firm) and to ATP’s program emphasis on joint ventures.

Cohen and Walsh defined appropriability as “the degree to which different appropriability mechanisms, such as secrecy, patents, or the exploitation of first mover advantages, increase the rents ¹⁹⁶ due to R&D....” ¹⁹⁷ Flows of R&D-related information, in their study, have two offsetting effects on a firm’s interest in investing in R&D. On the one hand, information flow diminishes appropriability, and thus dampens incentives to conduct R&D. On the other hand, it increases the R&D productivity of the firms that receive spillover flows, and in turn increases the productivity of R&D conducted at the industry level.

The diagram in Figure 7–1 illustrates the hypothesized relationships. In the researchers’ words:

To the degree that R&D related information flows are stronger within an industry, the more difficult appropriation of rents to R&D will be, notwithstanding the particular appropriability mechanism employed. However, use of different appropriability mechanisms may, at the same time, diminish the extent and value to rivals of intra-industry information flows. For example, to the degree that secrecy is used and is effective, we would expect information flows to be less than if patents are used since patents, while offering protection, also disclose information. (p. 9)

Figure 7–1. Relationships across Industry R&D Intensity, Intra-Industry R&D Information Flows, and Appropriability Mechanism Effectiveness

Source: Cohen and Walsh, R&D Spillovers, Appropriability and R&D Intensity: A Survey Based Approach, 2000, p. 44.

Cohen-Walsh Method of Isolating Knowledge Spillover Effects on Productivity

In their study, Cohen and Walsh constructed a simultaneous equation model that expressly links the dependent variables—R&D intensity, appropriability, and information flows—to firm and industry level economic variables. They identified “R&D intensity” as the sales-weighted average of the R&D intensities of business units in each industry.

Their measure of appropriability was based on the responses of firms to six appropriability mechanisms, including secrecy and patents. Their measure of information flows, which was also taken from survey data, reflected the percentage of respondents in an industry reporting that information from rivals suggested new R&D projects.

Given their specification of five dependent variables, Cohen and Walsh specified a set of five simultaneous equations to determine them. One equation expressed R&D intensity as a function of appropriability and intra-industry R&D information flows, controlling for market-mediated information flows, information from suppliers, generic science base, and demand growth. The second equation expressed information flows as a function of industry R&D intensity and appropriability, controlling for market-mediated information flows, number of technological competitors, and extra industry information from suppliers, customers, and universities. The third, fourth, and fifth equations expressed three aspects of the appropriability mechanism. The researchers used two- and three-stage least squares estimation techniques to solve the equations.

Testing the Model with Data from the 1994 Carnegie Mellon Survey on Industrial R&D

The researchers tested the model with data from an extensive, existing database constructed from a 1994 survey of industrial R&D in the United States. Referred to as the Carnegie Mellon Survey, the mail survey was sent to R&D unit directors for manufacturing firms. ¹⁹⁸ A distinguishing feature of the data from this survey is that it provides separate measures of appropriability and intra-industry R&D information flows, thus permitting “control for the effect of intra-industry R&D information flows on appropriability,” thus, in turn, making it possible to “observe the possibly countervailing effect of these flows on R&D itself.” ¹⁹⁹ Cohen and Walsh obtained other firm- and industry-level data for their study from standard sources (e.g., COMPUSTAT and Census of Manufacturers’ (1992) special surveys).

Study Findings

Study findings suggested that “...the direct influence of intra-industry R&D information flows is strongly complementary to R&D at the industry level...” ²⁰⁰

Overall, the study’s key findings were that:

• The more appropriable are the rents to R&D, the higher the R&D intensity of an industry;
• There is a negative relationship between selected appropriability mechanisms (particularly secrecy) and intra-industry R&D-related information flows; and
• Intra-industry R&D-related information flows lead to greater R&D intensity.

The study findings indicated that, “…controlling for the effect of intra-industry information flows on appropriability, intraindustry R&D information flows complement firms’ own R&D efforts, underscoring the social welfare benefits of such flows.” ²⁰¹ These findings are consistent with fundamental propositions leading to ATP’s establishment. They also point to the possibilities of industrywide as opposed to firm specific benefits from ATP awards.

The study’s limitations are candidly noted, and indeed may be seen as representative of the problems and limitations encountered in econometric work. The limitations include considerable measurement error in the survey-based measures, the ad hoc character of some model specifications for which there is little theory to offer guidance, the opportunistic character of some model specifications driven mainly by the availability of data, and the lack of robustness of a number of findings across model specifications and estimation methods.

Analyzing the Role of Universities in Public-Private Partnerships

Bronwyn Hall, University of California, Berkeley and National Bureau of Economic Research (NBER), Albert Link, University of North Carolina-Greensboro, and John Scott, Dartmouth College, used Probit and Tobit estimators to analyze survey results to explore the role of universities as research partners in ATP-funded projects. ²⁰² They were dealing with a research question that entails a dependent variable that, like the halo effect, is not directly observable, thereby requiring the use of the non-standard estimators.

What Role do Universities Play in Research Partnerships?

The study explored the research role universities play in ATP-funded projects at three levels. First, the researchers simply looked at the organizational role the universities had in the various projects—either as a research partner or as a subcontractor. Second, they explored the research role played by universities by asking project representatives to respond to a statement that “this research project has experienced difficulties acquiring and assimilating basic knowledge necessary for the project’s progress.” To examine the responses to the statement more systematically, they estimated Probit models to explain inter-project differences in responses to the statement. ²⁰³

Hall, et al., made the following observations based on their results:

• Respondents with a university participant (as a research partner or as a subcontractor) systematically agreed that their projects experienced difficulties acquiring and assimilating basic knowledge necessary for progress toward completion.
• Prior experience working with a university as a research partner or as a subcontractor is a very significant factor in decreasing the difficulty of acquiring and assimilating basic knowledge.
• Acquisition and assimilation difficulties with basic knowledge decrease slightly as overall project size increases.
• Projects in the electronics area have substantially more difficulty in acquiring and assimilating basic knowledge than do projects in other technology areas. (p. 18.)

Do Universities Enhance the Research Efficiency of Research Partnerships?

Descriptive statistics from the survey did not provide a clear answer to the question of whether universities enhance the research efficiency of research partnerships.

The descriptive statistics were based on survey responses to a series of three statements about unexpected research problems encountered, and to a series of twostatements about the productive use of complementary research resources. In the absence of a clear response pattern to the survey questions, the researchers examined the responses more systematically using Probit models. However, in this case the Probit models also did not show any significant, identifiable effects of universities on the efficiency of research partnerships. The researchers concluded that the presence of unexpected problems is either a random event or too complex to disentangle using their approach. ²⁰⁴

Do Universities Affect the Development and Commercialization of Industry Technology?

To address this question, the researchers asked survey respondents to respond to two statements. One of the statements regarded the generation of new applications of the technology over the course of the project. Survey results showed conflicting results between joint ventures and single applicants. Using Probit estimates also produced insignificant results regarding university influence on the generation of new applications of technology developed in projects.

The second statement concerned faster-than-expected commercialization of the technology. Survey statistics showed single applicants with no university involvement to be the most optimistic about accelerated commercialization. The Probit estimates shed further light on the relationship between university involvement and technology commercialization. Projects involving universities as partners were found to be less likely to develop and commercialize technology sooner than expected. The researchers speculated about possible explanatory factors. ²⁰⁵

Is There a Relationship Between University Involvement and Project Termination?

Hall et al., also investigated the relationship between university involvement in an ATP-funded project and the probability that the project will terminate early, using the following Probit model:

Probability that project i terminates early = F(Xi b ),

where F is the cumulative normal probability function, and Xi is a vector of variables that characterizes project i . The variables are ATP’s share of funding, involvement of a university, type of project, size of the lead participant, technology area, and a time variable denoting the year in which each project was initially funded.

The model was applied to the analysis of 21 projects that had terminated prior to completion. The group of terminated projects included 11 joint ventures, three of which included a university as a research partner and two others that included a university as a subcontractor, and 10 single companies, 4 of which included a university as a subcontractor. Hence, of the 21 projects in the group 9 involved a university and 12 did not.

Hall et al., calculated Probit estimates using alternative specifications of the above equation. Using the results of the Probit analysis to control for possible sample selection bias, they concluded that the calculated probability of early termination is lower for each discrete level of ATP’s share of funding when a university is involved in the project. ²⁰⁶

Modeling the Impact of Publishing by Industry Scientists on the Quality of Innovative Output

Bruce Kogut, University of Pennsylvania’s Wharton School, and Michelle Gittelman, New York University’s Stern School of Business, carried out an econometric/ statistical study to help clarify the relationship between publishing by a firm’s scientists and the firm’s innovative output as indicated by citations of its patents. ²⁰⁷ Their approach was based on a growing body of empirical research provided by Trajtenberg, Harhoff, Narin, Sherer, Vopel, Lerner, Shane, and others. That body of research indicates that “patent citations contain information about a patent’s technological importance, and that they can also be used as a proxy for economic value.” ²⁰⁸ The study is relevant to ATP in that it bears on the appropriability of returns of publicly funded research, and on the importance of collaborations to realize the benefits of research in commercial applications.

Exploring Two Questions about the Relationship of Publishing and Patenting

The study examined firm experience in the biotechnology area of human therapeutics, an important area for ATP project funding. Kogut and Gittelman selected the biotechnology area as an industry in which “the productivity of a firm’s R&D investments can be greatly improved by incorporating scientific research into the firm... a process that involves the transformation of codifiable knowledge into tangible goods, services and technologies.” ²⁰⁹ Figure 7–2 illustrates how, in the emerging field of gene therapy, acceleration of the rate of publishing seems to lead to acceleration of the rate of patenting.

Figure 7–2. Patenting and Publishing in the Scientific Literature, Gene Therapy: Patents and Publications

Source: Kogut and Gittelman, Public-Private Partnering and Innovation Performance Among U.S. Biotechnology Firms, 2001, p. 4.

Kogut and Gittelman focused on answering two questions: (1) Do investments in scientific research, as proxied by publications, pay off for the firm in terms of producing valuable firm-level research capabilities, as proxied by patents? (2) Does the firm’s research capabilities and collaborations, as proxied by co-publications, affect the quality of commercial innovation, as measured by patent citations? (Their measure of the quality of innovative output was based on the number of citations of the firm’s patents by other firms in subsequent patents.)

Kogut and Gittelman had a special interest in the effect of industry-based scientists co-publishing with university-based scientists, noting that, “Direct linkages between firms and universities are likely to be an important organizational arrangement by which research-oriented firms extract gains from the complementarities between scientific research and technological innovation.” ²¹⁰ When ATP-funded projects include university participation, opportunities for co-publishing between firm scientists and university scientists are increased. Understanding the implications of stimulating this form of collaboration is relevant to ATP project selection.

Approach to Addressing the Questions

The general approach used by Kogut and Gittelman was to develop models of patenting and publication, develop a sampling strategy, compile data, and apply the models to address the two questions.

Because of several sampling problems, Kogut and Gittelman decided against selecting the data sample primarily from ATP firms. First of all, many of the ATP biotechnology patents were too recent to support use of citation analysis as a performance measure. Second, insufficient time had passed to build evidence of co-publishing/partnering relationships. And finally, Kogut and Gittelman felt that patenting in ATP-funded biotechnology projects was not representative of the importance of patenting as a means of protecting assets in other areas of biotechnology. ²¹¹

For these reasons, Kogut and Gittelman drew an initial sample of firms from U.S. biotechnology firms at large. This sample included 114 firms, of which seven of the companies were participants in ATP-funded projects. They then added 39 companies with ATP-funded projects, for a total of 153 companies in the sample.

Kogut and Gittelman collected four sets of data for the sample firms: data on publications in scientific literature, patent data, scientist data, and data for firm-level characteristics. ²¹² They identified 20,477 publications by authors in the sample; and 2,269 patenting authors, of which 35% were also listed on a publication. ²¹³

Kogut and Gittelman developed several predictive models, the dependent variable of which was the cumulative forward citation frequencies to all patents in a patent family. The explanatory variables include firm effects, scientist effects, firm-level controls, and patent-level controls.

Applying negative binomial regression as their estimation technique, Kogut and Gittelman obtained firm and patent-level measures from four models they developed. The first model included only control variables, the second model added the log of the number of publications by the firm up to the year of the observed patent, the third model included only firm-level publishing variables, and the fourth model added patent-level measures. Kogut and Gittelman used the models separately to measure firm effects and patent-level effects.

Kogut-Gittelman Findings

Kogut and Gittelman concluded, “Research capabilities and the innovative capabilities of biotechnology are linked through complex, indirect pathways.” ²¹⁴ Specific findings from the study were as follows: ²¹⁵

• Research-intensive firms contribute new knowledge to the public domain through scientific publications.
• Collaboration with universities is a central mechanism by which biotechnology firms acquire knowledge and disseminate research findings.
• Larger firms account for the greatest volume of scientific outputs.
• Younger and smaller firms tend to publish at the forefront of discovery.
• Younger and smaller firms produce the most highly cited articles.
• Collaboration plays different roles for firms with different research capabilities.
• Firms that lack strong in-house research capabilities improve their research performance by collaborating; firms with strong research capabilities collaborate just as much as weak ones, but collaboration does not improve their research performance.
• The more a patent builds on scientific literature, the more it is cited.
• Good patents build on science, but firms that publish a lot of good quality science do worse in producing highly cited patents.
• Investment in firm-level scientific research may be necessary to allow the firm to translate scientific findings into innovative projects, but the actual outputs of that research—contributions to the scientific literature—do not appear to pay off directly on the commercial side.

Kogut and Gittelman’s findings have implications for ATP in that they reinforce the argument that firms find it difficult to appropriate the returns to scientific research placed in the public domain. Though firms nevertheless gain from performing research, they need to do more than publish; they need direct connections with those who translate knowledge into commercial applications. “Public-private partnering is a key organizational arrangement by which this process occurs,” according to the authors. ²¹⁶

Investigating Characteristics and Impacts of Joint Ventures

ATP has funded several econometric/statistical studies to assess the impact of joint ventures and the characteristics of the partners to the joint ventures. These studies encompass many of the issues of data selection and collection, model specification, and interpretation that are typical of econometric studies.

The logic of private sector joint ventures, in large part, relates to the sharing of R&D costs, the spreading of risks, the exploitation of scale economies in R&D, the access to new capabilities among several firms, and supply-chain linkages to accelerate technology development and commercialization. For the public sector to support these joint ventures, the logic expands to include the generation and utilization of spillovers, ²¹⁷ and the fostering of joint venture formations.

As noted by David Mowery, University of California, Berkeley, Joanne Oxley, University of Michigan, and Brian Silverman, University of Toronto, in their report, “Economists have long argued that consortia have the potential to internalize spillovers of technological knowledge among firms, thereby reducing disincentives to invest in R&D and encouraging more rapid diffusion of technology among firms.” ²¹⁸ They also note that the effects of consortia on the incidence of spillovers “have received very little empirical attention,” and further, “[a] better empirical framework for such analyses is essential to effective evaluation.” ²¹⁹

Study of Spillovers and Consortia

The Mowery et al., study addressed several aspects of the relationship between spillovers and consortia (or joint ventures). ²²⁰ In particular, the study investigated the influence of inter-firm knowledge spillovers on ATP’s selection of awardees, and the extent to which an ATP award affects the level of spillovers among members of a joint venture. In each case, a set of prior theoretical propositions was used to formulate a specific testable hypothesis.

Three Hypotheses

The study set forth the following three hypotheses:

H1: If ATP-supported consortia favor the internalization of knowledge spillovers among member firms, the probability that ATP will fund a consortium proposal should increase as the level of patent cross-citation among this group of firms increases.

H2: Firms participating in a consortium will cite one another’s patents more heavily following the formation of the consortium.

H3: Firms participating in a consortium will tend to cite one another’s patents more rapidly after the issuance of these patents than was true prior to the formation of the consortium. (p. 6)

Mowery et al., noted that knowledge spillover benefits should apply equally to horizontal and vertical consortia, and, in fact, most consortia involve both horizontal and vertical elements. They also acknowledged that the conceptual framework underpinning H1 “…ignores other possible influences on the formation of consortia.” ²²¹

They pointed to theoretical propositions suggesting “...consortia are likely to be more effective when they involve as members firms that generate a high level of spillovers among themselves in the absence of a consortium.” ²²² The second and third hypotheses related to the post-formation behavior of a consortium in terms of its effectiveness in generating knowledge spillovers. The crafting of these hypotheses is of interest because the hypotheses detailed important aspects of evaluation design, including selection of the proxy variable, specification of a control group, and the role of trend and history as confounding explanations.

Patent Citations as a Measure of Spillovers

The measure of R&D spillovers used in this study—as in several other studies supported by ATP—is the citation of other firms’ patents in a firm’s own patent applications. Like the other researchers who have used this measure, Mowery et al., explicitly noted that patents are not a perfect measure of innovative output. Patents relate only to codified knowledge relevant to a new invention or technology. In addition, there are significant inter-firm differences in patenting behavior. Still, they noted that patents have certain advantages in empirical work: “All patents include a section devoted to citations of related patents, and these citations can be interpreted as a measure of inter-firm spillovers of knowledge.” ²²³

Also, patents are accessible in machine-readable form, which reduces the cost and increases the flexibility of using these data.

Constructing Portfolios of Patent Data for Testing the Hypotheses

To bound the empirical work, Mowery et al., compiled patent-related data for four groups of consortia: (1) semiconductor-related patent data compiled from members of the SEMATECH Research Corporation for patents mainly derived from SEMATECH research (SEMATECH, a consortium founded in 1987, provides a longer history than ATP); (2) control groups of patent data from SEMATECH members not derived from SEMATECH research, from U.S. firms that are not members of SEMATECH, and from U.S. universities and federal laboratories (the SEMATECH and control group data were compiled for the 1985–1995 period); (3) patent data compiled for participants in ATP-funded joint ventures; and (4) patent data compiled for unsuccessful joint venture applicants to ATP. The researchers also compiled a “complete dataset on the ownership of all firms in the sample, in order to minimize spurious results from the analysis of patent cross-citations among these firms...” ²²⁴

Study Design

The study design called for a before-and-after SEMATECH analysis, a before-and- after ATP analysis, and a comparison of the patenting experience of SEMATECH member firms with the before-and-after experience of consortia that did not receive federal funding. The research team, however, stated that it was not able to construct a control sample of consortia for which it was certain no public subsidies had been received, and thus did not conduct the latter comparison.

The purpose of the SEMATECH analysis was to test the analytical approach proposed for H2 and H3 for ATP joint ventures, and to provide additional insights about the hypothesized relationships. The researchers believed SEMATECH’s longer patenting history offered a better proving ground for the approach than would ATP.

With respect to the testing of H1, which pertained to the influence of pre-application cross-citations on ATP’s selection of a joint venture for funding, the underlying model was as follows: The dependent variable, SUCCESS (which was given a value of 1 for successful applications to ATP and 0 for unsuccessful applications), was treated as a function of patent cross-citations, PCROSS, which was defined as follows:

PCROSS = · [ (citationi, j ­ i/total citesi)]/N

where firm i = the citing firm, firm j = the cited firms in the same group, and N = the total number of firms in the group.

As the researchers acknowledged, they lacked a full set of controls for potential influences on ATP funding decisions. However, they did control for a number of factors, including: the number of firms in a given application to ATP, the level of experience collaborating, whether the application was to an ATP focused program competition or a general competition, and the average number of patents per firm in the application. ²²⁵

With respect to H2 and H3, which pertained to the extent and speed of inter-firm cross-citations following establishment of the consortia, the researchers focused on SEMATECH. They used a modified approach to test H1 for SEMATECH to reflect the fact that they were working with only one consortium. For 10 SEMATECH firms, they matched each SEMATECH member firm with a nonmember firm from the control group, acknowledging that the control group firms were imperfect matches. The researchers compared the number of relevant patents assigned to each from 1975–96, and the number of citations of these patents. They tested a modified version of H1 for SEMATECH.

Mowery et al., Findings

The researchers concluded that the preliminary results of their testing of the hypotheses using SEMATECH data “are not especially encouraging.” With respect to H1, they concluded: ²²⁶

...although SEMATECH firms display relatively high levels of knowledge spillovers among themselves prior to the formation of the consortium, we find no evidence that these spillovers have increased since SEMATECH’s establishment, nor do we find that member firms have been able to exploit one another’s patents more rapidly since the establishment of the consortium. (pp. 26–27)

In testing H1as it applied to ATP, the researchers concluded: ²²⁶

Although they omit some important influences on ATP selection outcomes, these results suggest either that patent cross-citations only weakly proxy the “true” inter-firm knowledge spillovers that may influence the outcome of ATP consortium funding competitions, or that such pre-consortium spillovers exert a relatively weak influence on ATP’s awards. (p. 13)

In view of this finding suggesting little evidence that ATP funds are deliberately directed toward internalizing spillovers among firms with substantial pre-ATP spillovers, the researchers debated the desirability of this policy orientation. They concluded that, indeed, it might be desirable, based on related prior findings suggesting that firms with high levels of cross-citations might not require public subsidies to encourage them to form consortia. In the words of the researchers:

In this view, publicly funded consortia should involve firms that exhibit a level of pre-consortium knowledge spillovers that is lower than the level of spillovers observed among firms that form unsubsidized alliances or consortia. (p. 14)

The researchers pointed out that, on the other hand, consortium performance is probably improved by some degree of “technological overlap” that helps mutual understanding and absorption of research results. Mowery et al. concluded that a larger sample of collaborative ventures is needed to improve the strength of their results.

Darby et al., Study of Joint Venture Effects on Participating Firms

Michael Darby and Lynne Zucker, University of California, Los Angeles and the National Bureau of Economic Research and Andrew Wang of ATP also used econometric/statistical techniques to study the effects on firms participating in ATP-funded joint ventures. ²²⁷ The Darby et al., study focused on comparisons between single firm and joint venture projects. For each type of project, the researchers investigated the role of university participation either as a full member in a joint venture or as a subcontractor for a single firm or joint venture project. The basic hypotheses of the study were that participation in a joint venture should increase the patenting rate of participating firms, and that the effect should be larger if the firm has a university partner or university subcontractor.

Study Design

The study design entailed a before-and-after comparison of innovation outcomes, using the firm/organization as the unit of analysis and patent data as the key indicator of impact on firm innovation. In fact, the researchers stated, “Patents, in representing an active business decision to protect commercially valuable inventions, are arguably the single best proxy measure of innovation.” ²²⁸ They tracked patenting by ATP awardees before, during and after they became ATP participants. A hierarchy of settings also was constructed, indicating whether a firm was a full partner in a joint venture or a single firm participant, and, given this initial classification, whether or not it had a university as a collaborator or a subcontractor. The time period covered by the patent series was 1988–1996.

The authors developed a “deflated” patent count measure for their dataset, which is adjusted for year-to-year changes in the average rate of patenting as measured by patents per assignee for all U.S. assignees of U.S. patents. This adjustment was necessary to reflect national trends in the rate of patent applications and the speed with which patents were processed. (“A simple before-and-after comparison of patenting is therefore subject to the criticism that it reflects trend increases in patenting rather than identifying real program impact.”) ²²⁹

Darby, et al. Findings

In a cross-section before-and-after comparison of each firm that participated in ATP projects that started by the end of 1995, the authors controlled for firm size, industry, total amount of ATP funds received, and year of participation. Results from a series of regressions based on these controls indicated that both single firm and joint venture participants increase their patenting when they have university partners or university subcontractors.

Although the magnitude of the impact varied across model specifications, the authors concluded, that before-and-after patenting rates generally increase after ATP participation under a number of different program and participant variations. “For firms in the sample, patenting increased on average by between 5 and 30 patents per year during the period of participation. These estimates are conservative since future effects from the ATP project participation are not included, even though they are implied in [the] regression models.” ²³⁰ The estimated impact on patenting rates was higher for firms that participated in joint ventures than for single firms, and for both sets of those firms that had university partners.

Changing the “Social Embeddedness” of Firms

The Darby et al., study offers additional a priori justification for the emphasis of ATP’s program design on joint ventures and the participation of universities as R&D performers. The researchers credit the ATP awards with changing the “social embeddedness” of participants in networks of relationships with other firms and other organizations. ATP is thus seen not only as providing awards to participants but also “fostering institutions and social processes that facilitate innovation.” ²³¹

Sakakibara-Branstetter Impact Study of Joint Ventures

The Mowery et al., and Darby et al., studies examined the effects of firm participation in joint ventures in a before-and-after mode of evaluation. Mariko Sakakibara, UCLA, and Lee Branstetter, Columbia Business School, also used an econometric/statistical approach to modeling the effects of firm participation in joint ventures, but took a different angle from these other studies. ²³²

Sakakibara and Branstetter examined the impact of ATP-funded consortia on the ex post research productivity of participating firms. Their thesis was that if participation in an ATP-sponsored consortium increased the research productivity of member firms by “promoting research spillovers among members,” and allowing for exchange of complementary knowledge assets, then there should be a statistical relationship between the number of ATP consortia a firm is involved in during a given year, and the firm’s patent output in that year. ²³³ The authors noted that a lag structure should be introduced, but cited the limited time-series dimension of their U.S. data as a reason for not building lags into several of their estimation equations. They tested their hypothesis using panel data on participating firms and a control group of non-participants.

Three Research Questions

The study was subdivided into three research questions of growing econometric complexity: (1) To what extent does participation in an ATP research consortium contribute to an overall expansion of research productivity among participating firms? (2) What impact does participation in research consortium have on the collective patenting of participating firms in the technological areas targeted by the consortia—within which is nested the further question of what kinds of consortia are the most successful at promoting the ex post research productivity of participating firms? (3) How are benefits from participation in an ATP-financed consortium distributed by type of firm?

Testing the Model with Japanese Data

As explained in Chapter 4, the study used data from publicly funded Japanese research consortia as a “statistical ‘testing ground’ for the analytical frameworks that were applied to U.S. data…” ²³⁴ An important aspect of this framework was that it highlights the importance of the length of time covered by the data series. Japanese public sector support of R&D consortia dates back to the 1950s, and Sakakibara and Branstetter reported construction of a dataset for Japan that goes back to the early 1980s. ATP projects were begun in 1990–1991, with only a small number of projects funded until the mid-1990s. However, public databases on patent information and R&D spending available to the researchers only extend to 1995 just as ATP was expanding its support of research consortia. ²³⁵ This difference in length of data series is important, according to the researchers, because:

...results from Japanese data suggest that much of the impact of research consortia is felt long after the inception of the project. In fact, evidence from the Japanese consortia suggests that some of the strongest effects are felt after the official cessation of the consortia. (p. xiv, emphasis in the original)

Applying the Model to ATP Data

Sakakibara and Branstetter drew from their previous work ²³⁶ to construct a model to determine if there is an observable statistical relationship between the intensity of participation in an ATP-funded joint venture by a firm and the firm’s patent output in that year. They acknowledged that the role of lags is of interest in addressing this question, but they stated that the “limited time-series dimension of our data does not allow us to adequately explore this question, but we do introduce a lag structure in subsequent empirical sections.” They used the following log-linear equation to address the question:

r it = ß 0 + ß 1 r it + ß 2 C it + S d d D id + µ it

where r it = natural log of the number of patents generated by firm i in year t ; r it = the natural log of firm-level R&D spending; C it = the intensity of participation in research consortia, measured as the count of concurrent projects in which firm i was involved in year t ; d ’s = the coefficients on the industry dummy variables ( Ds ); µ = an error term; and d terms are industry-level differences in the propensity to patent. ²³⁷

U.S. data used in the estimation equation consisted of panel data for 249 firms, 65 of which had participated in at least one ATP project; the data covered the years 1985–1995. The panel was unbalanced in that participant firms tended to be larger, conducted more R&D, and generated more patents than the nonparticipant sample. ²³⁸

Sakakibara-Branstetter Findings

The study found that “Controlling for firm size, impact of participation in ATP-funded consortia on research output remains positive and significant in all specifications.” ²³⁹ In quantitative terms, the researchers concluded, “participation in one additional ATP-funded research consortium per year would generate an increase in patenting in that year of nearly 8%.” ²⁴⁰ They also cited evidence that “large firms conducting intensive research and development (R&D) tend to benefit more from their participation in consortia” ²⁴¹ but prudently cautioned against jumping to conclusions based on these results because of the several difficulties the study encountered in constructing a reasonably lengthy time series of data and a representative comparison group.

A major limitation of the study, as noted above, was the truncation of the time series dimension of the U.S. data. Thus, the researchers believed it is likely that a large share, perhaps the largest share, of the benefits from participation in ATP was missing from the data available at the time they performed the study. Difficulties were also encountered in obtaining data on the smaller firms that are involved in ATP-sponsored consortia. In their words:

While it is relatively easy to obtain data on the larger, publicly traded firms that were part of these projects, it is difficult to obtain similar data on small, privately held firms that were often leaders of the joint ventures. Lack of data on smaller firms limits our ability to estimate the impact of ATP-funded consortia on their research productivity. If smaller firms benefit more from consortia participation than do larger firms, then our data may underestimate the impact of ATP-funded consortia on smaller firms. (p. xv)

Sakakibara and Branstetter found that ATP’s confidentiality agreement constrained their access to data. They noted that participating firms submitted a substantial amount of information to ATP through the Business Reporting System (BRS) (see Chapter 5.) Because the information submitted through BRS is divulged under an agreement between the firms and ATP to maintain the confidentiality of the information, they were not able to examine firm’s individual responses to survey questions about the impacts of various projects. ²⁴²

Emphasis on Data

The attention to data compilation and quality is a singular feature of the Sakakibara-Branstetter study. Indeed, in noting that they have provided ATP with full documentation of the database used in the study, the authors expressed their hope that “these tables and documentation will be a useful, enduring data resource for ATP.” (p. xviii)

Summary of Econometric/Statistical Methods

By the end of ATP’s first decade, use of the econometric/statistical method constituted an increasingly important component of ATP’s evaluation program. While many of the studies were exploratory in nature, use of econometric/statistics methods contributed to ATP’s ability to test hypotheses about underlying relationships and answer demanding questions about the program’s net contribution, and what might have happened in the absence of the program. By embedding procedures such as before/after comparisons and the use of control groups in research design, the methods were used to control for alternative explanations of observed behavior. They have also been used to strengthen statistical tests of the significance of effects not directly observable, such as the halo effect, and to identify characteristics associated with awarded projects. Econometric/statistical studies have also helped define and clarify key ATP program theory concepts, such as the impact of the program on firm productivity.

As noted, several of the studies surveyed in this chapter dealt with measures of spillovers, a key justification for public sector support of firm-specific R&D, but a concept not widely used outside economics. Despite their limitations, these studies have provided a methodologically more consistent, replicable, and convincing form of evidence about program impacts than the evaluation methods described in earlier chapters.

The findings of the group of studies presented here are as much about the existence of positive impacts (e.g., the presence of spillovers) as they are about the absence of negatives (e.g., that ATP funding does not displace private-sector R&D funds). The findings are rich in new insights about the program. They also help confirm concepts suggested by other methods, such as the halo effect. They point to differential impacts of ATP support depending on the structure and social context of joint ventures, and depending on the presence or absence of universities as project participants or subcontractors. Obviously, information of this type can inform those charged with assessing proposals against ATP’s selection criteria.

As emphasized throughout this chapter, econometric/statistical methods involve extensive efforts at creating primary datasets and the refinement of existing datasets. Thus, the studies provide a foundation for subsequent work by others in examining multiple dimensions of the program. The attention given to data collection by the studies highlights the fact that use of econometric/statistical methods, particularly at the initial phases of a program’s operations, requires major up-front investments in data construction. It confirms a point made in Chapter 3: that econometrics is as much about identification, collection, and cleaning primary data as it is about model formulation and statistical techniques. The attention to data collection, in addition to model development, also reinforces the observation that ATP’s evaluation program generates its own spillovers. By supporting primary data collection, ATP provides a common pool resource that can be used by others studying ATP’s impacts as well as by researchers and decision makers interested in generic processes of technological innovation or of joint ventures.

The limits of the methods are also evident in this collection of studies, especially as several authors were quite candid about the ad hoc character of some of their model specifications, and the tentative, qualified character of their findings.

Several researchers, in fact, pointed to the atheoretical formulation of their estimation equations and their findings’ lack of robustness, meaning that findings about impacts were heavily dependent on the formulation of a model. Several studies also noted problems associated with data collection. The chief problem was the short period for which public data were available (e.g., patent data), thus truncating analysis of long-term impacts. Other data problems included difficulties in constructing balanced comparison groups, limited access to confidential ATP records, and the heavy dependence of several of the studies on patent data. Although patent data are widely used in econometric studies of the impacts of R&D, they are certainly not the only measure of the economic or inter-organizational impacts of R&D, as several of the authors noted. Indeed, they may not be the best measure.

Finally, compounding the force of these limitations in terms of the acceptability of econometric/statistical evaluation methods is the lack of transparency. Noneconometricians may not see how findings were generated. The use of econometric/ statistical methods is not only complex, but also typically involves a large number of assumptions. Decision makers’ uncertainty about those assumptions and the inner workings of the formulated models can deter acceptance of findings. Yet, as noted earlier, despite these limitations, econometric/statistical methods enable researchers to formulate and test critical hypotheses in ways that the other methods already discussed do not.

____________________
183 Feldman and Kelley, Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect, 2001.

184 In order to deal with a research question where the dependent variable of the structural estimating model is not directly observed—as is the situation in the case of the purported halo effect—standard econometric estimators are not suitable, and special estimators are needed. These include the Tobit and Probit estimators, as well as others. Econometric computational software is available for applying multivariate analysis, including logistic regression and the Tobit and Probit estimators. For additional explanation of the Tobit and Probit estimators, and software for their application, see the TSP User’s Guide, which is publicly available for viewing and downloading at the website of the University of California, Berkeley’s Econometrics Laboratory, the Software Archive (see http://elsa.berkeley.edu).

185 Ibid.

186 Griliches et al., R&D Policy in Israel: Overview and Lessons for the ATP, 2000. (See Chapter 4 for previous discussions of this work and Chapter 9 for discussions related to the study’s findings.)

187 Ibid., p. 9.

188 Ibid., p. 8.

189 Ibid, p. 8, footnote 3.

190 Ibid., p. 8.

191 Ibid., p. 8.

192 Ibid., p. 10.

193 Ibid., p. 10.

194 Ibid., p. 11.

195 Wesley M. Cohen and John P. Walsh, R&D Spillovers, Appropriability and R&D Intensity: A Survey Based Approach, Draft report, ATP, 2000.

196 Rent in this case refers to a return in excess of a competitive rate of return.

197 Ibid., p. 9

198 Wesley Cohen and Richard Nelson conducted the survey.

199 Ibid., p. 5.

200 Ibid., p. 20.

201 Ibid., p.6.

202 Hall et al., Universities as Research Partners, 2002.

203 As explained earlier, in some circumstances standard econometric estimators are not suitable for testing hypotheses, and special estimators are needed.

204 Ibid., pp. 18–19.

205 Ibid., pp. 22–26.

206 Ibid., pp. 11–13.

207 Bruce Kogut and Michelle Gittelman, Public-Private Partnering and Innovation Performance Among U.S. Biotechnology Firms, Draft report, ATP, 2001.

208 Ibid., p. 26.

209 Ibid., p. 2.

210 Ibid., p. 2.

211 Ibid., p. 8.

212 Ibid., p. 9.

213 Ibid., pp. 12–13.

214 Ibid., p. 35.

215 Ibid., pp. 33–35.

216 Ibid., p. 35.

217 “Among the hypothesized benefits of consortia, knowledge spillovers provide the strongest motivation for public investment in their formation and operation,” according to David C. Mowery, Joanne E. Oxley, and Brian S. Silverman, The Role of Knowledge Spillovers in ATP Consortia, Draft report, ATP, 1998, p. 2.

218 Ibid., p. i.

219 Ibid., p. i.

220 For the purpose of this discussion, the terms “consortia” and “joint venture” are used interchangeably.

221 Ibid., p. 13.

222 Ibid., p. 2.

223 Ibid., p. 5.

224 Ibid., p. i.

225 Ibid., pp. 8–9.

226 The researchers concluded that it was premature to test H2 and H3 using ATP data.

227 Michael R. Darby, Lynne G. Zucker, and Andrew Wang, Program Design and Firm Success in the Advanced Technology Program: Project Structure and Innovation Outcomes, NISTIR 6943 (Gaithersburg, MD: National Institute of Standards and Technology, 2002).

228 Ibid., p. 6.

229 Ibid., p. 10.

230 Ibid., p. 19

231 Ibid., p. 20.

232 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002.

233 Ibid., p. 6.

234 Ibid., p. xiii.

235 Ibid., pp. xiv-xv.

236 Lee Branstetter and Mariko Sakakibara, “Japanese Research Consortia: A Microeconometric Analysis of Industrial Policy,” Journal of Industrial Economics, 46: 207–233,1998.

237 Sakakibara and Branstetter, Measuring the Impact of ATP-Funded Research Consortia on Research Productivity of Participating Firms: A Framework Using Both U.S. and Japanese Data, 2002, p. 5.

238 Ibid., pp. 7–8.

239 Ibid., p. 13.

240 Ibid., p. 13.

241 Ibid., vi.

242 Ibid., see p. xv.

Return to Table of Contents or go to Chapter 8: Other Evaluation Methods Used by ATP .

Date created: July 22, 2004
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)