A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

CHAPTER 5:  Survey Method

As discussed in Chapter 2, a survey, if carefully designed, collected, and administered, can reveal useful information about a program’s development long before impacts can be discerned in other ways. A survey can provide information about the characteristics and activities of performers, early effects, and long-term expectations. This information can help program managers detect early accomplishments and early problem areas. It can also be of value in communicating with Congressional committees and other key stakeholders who want to know if a program is on track to achieve its defined goals.

This chapter describes how ATP’s use of the survey method evolved, allowing evaluators to track the progress of ATP projects, to present aggregate statistics in meaningful ways, to gather data for case studies, and to shed light on questions of critical importance. The surveys described here range from simple to elaborate, from relatively inexpensive to costly, and from those performed by outside contractors to those performed in-house. They range from those that are relatively straightforward to analyze, to those that are intricate and complex and contain many levels of data. Further, the surveys presented here demonstrate the varied purposes served by this useful tool of analysis, and show how a program can build its expertise in the use of a method over time. Table 5–1 lists eight ATP commissioned studies that have used survey as a method of research. The table indicates the main purposes of each survey, its approach, and any unique features.

Table 5–1. Eight of Eleven Studies Using Survey Method Represented*

*Note: Three additional studies by Powell, listed in Tables 3–2, 3–3, and 3–4, detailing development and use of the BRS survey are not separately listed here, as the material is covered adequately by the studies cited.

Gaining Early and Broad Perspective of a Program’s Effects

At the earliest practicable time—as the program neared the end of its first year of operation in 1991—ATP commissioned a survey study of projects. The objective was to learn if the ATP awards were having identifiable effects and, if so, what they were. The impetus was two-fold: to aid internal program management and to answer stakeholder questions. Program administrators were eager for evidence that the new program was on track, and both supporters and detractors in the public policy community were looking for quick answers to their questions.

The ATP engaged Solomon Associates, a small consulting firm, to work with ATP staff in designing and carrying out this first survey. ¹¹⁹ In selecting the target population, the contractor and ATP staff concluded that the multi-year projects funded by ATP should be up and running a minimum of 6–12 months before being surveyed to allow sufficient time for short-run effects to manifest themselves. Thus, ATP decided to proceed with the survey at the end of its first year of firstround projects.

At the time of this first survey, the newly formed program was in an experimental stage. It had a small staff, an overall budget of approximately $10 million, and a budget for external evaluation studies of $25,000. In addition to the survey, during the first year of program operation, ATP used administrative funds to support development of a database of program applicants and awardees. This database helped ATP answer stakeholder questions about the characteristics of its projects: the number, size, and affiliation of applicants; number of single company projects and joint venture projects proposed and awarded; geographical extent of the program as indicated by the location of applicants and awardees; technologies proposed and funded; amounts of funding requested and offered in industry cost sharing; and other characteristics of applicants and awardees. The descriptive profiling of applicants and awardees, together with the descriptive survey statistics of awardees after their projects were underway, constituted ATP’s initial evaluation effort.

Because only 11 projects were funded in the first year, it was an easy decision to include all of them in the survey—5 single company projects and 6 joint venture projects. Of the 35 organizations participating in the 11 projects, the survey was restricted to the 26 firms receiving ATP funding because, at that time, the program’s focus centered on firms rather than universities and other organizations. ¹²⁰ The sample included a range of firms in terms of the number of employees, sales volume, capitalization, and location.

Identification of a suitable respondent at each of the 26 companies in the survey was an iterative process involving ATP staff, the companies, and the contractor. Often the discussions led to the designation in smaller firms of the CEO as the respondent and the business contact or project’s technical program manager as the respondent in larger firms. Early in ATP’s evaluation program, then, the importance of correctly identifying the individuals to whom a survey is targeted was highlighted.

Because of the exploratory nature of this first survey, the survey designers decided to administer it by in-depth telephone interview, using an informal discussion style that followed a semi-structured interview guide and featured open-ended questions. The contractor, Samantha Solomon, conducted all of the interviews to preserve consistency and to provide the professional experience needed to pursue an open discussion that could accommodate unanticipated lines of inquiry while still covering the planned topics of the interview guide. While the topics covered by the interview guide were influenced by hypotheses about ATP’s intended effects (see Chapters 3 and 4), the open-ended nature of the line of questioning allowed respondents to bring up other topics and effects freely and to dismiss topics they thought unimportant for them. The objective was to discover any effects that the project participants might have experienced from ATP participation. ATP asked the contractor to allow the respondents free rein in identifying and discussing effects, and to code the responses after the fact, rather than to use a pre-coded format.

After a brief discussion in which the interviewer indicated general familiarity with the project, respondents were asked if they had experienced any changes as a result of the project and, if so, to discuss the single most important effect from their standpoint. Many respondents mentioned effects anticipated by ATP, but they identified two additional effects. In fact, respondents often listed one of those unanticipated effects, a “halo effect,” as the single most important effect of the award. The majority of the respondents—100% of the single company applicants —said receiving an ATP award bestowed a “halo” of enhanced credibility on the company (or words to that effect). This result heightened ATP staff’s awareness of the prestige value of the award, apart from the funding itself. It also had implications for the method of announcing awards and other operational aspects of the program. The results of this open-ended line of questioning clearly demonstrated the advantage of taking an exploratory approach to the survey method in the early stages of a program’s evaluation.

According to the contractor, respondents to the telephone survey tended to be emphatic in indicating when they thought a line of questioning was premature. They elaborated on areas of particular interest to them. Overall, they were described as willing and forthcoming in their discussions with the interviewer. There was 100% participation by the targeted respondents. ¹²¹

The resulting interview data showed some of the anticipated effects to be strong factors, others weak, and some too early to tell. Table 5–2 lists effects of the program identified by this first survey. The last two effects listed were those raised by respondents but not anticipated by the researcher. This first survey produced findings of considerable value to ATP managers. It showed that the projects were headed in the intended direction. It helped program evaluators and administrators better understand the types of impact the program was having and the relative timing of these effects. It identified some unexpected effects of a special nature, including the “halo effect.” The first survey also indicated which data were relatively easy or difficult to obtain, and areas of inquiry to which the companies were particularly sensitive. The feedback was invaluable in structuring ATP’s nascent evaluation program and, more specifically, in designing future surveys.

Table 5–2. Program Effects Treated in the First Survey of ATP Projects

Source: Solomon Associates, The Advanced Technology Program, An Assessment of Short-Term Impacts: First Competition Participants, 1993.

Extending and Deepening Survey Data on Program Effects

In 1993, three years after the program began, ATP initiated a much larger project survey that covered all 125 companies and consortia participating in the 60 projects funded from 1990 through 1992. The background obtained from the earlier survey, described in Chapter 5, offered insights in designing the larger survey.

Several options were considered in planning the second survey study. These options represented a tradeoff between survey continuity and adapting the survey to changing opportunities and issues. One option was to take a panel approach, whereby the original interview guide would have been used to re-interview the original group of respondents later in the project life cycles. This approach would have facilitated tracking emerging effects of the group of 26 companies in the original survey over the subsequent two years, but it would have been limited to this small, original group and the identical questions. Another option was to apply the original interview guide to a different group of respondents after one year of funding for comparability. This approach would have increased the size of the sample and allowed the comparison of progress for two different groups of funded companies at the same time in their funding histories. The results would have helped determine how representative were the results of the first survey, but it would not have allowed ATP to enlarge the scope of its inquiry.

Yet another option—the one taken—was to develop a new survey instrument and administer it to all projects that had received funding for at least six months. This approach had the advantages of allowing more extensive and in-depth coverage of a much larger sample. In addition to probing in greater depth each area of potential impact identified by the first survey, the second survey would explicitly ask about possible negative impacts. It would explore business goals, plans to commercialize new technologies, and progress towards commercialization. It would allow a comparison of progress over time, and would support analysis by size of company and by type of project (single company applicant versus joint venture). This latter survey approach was taken because the demand for quantitative information about the program’s performance as a whole had intensified by late 1993, particularly in light of discussions about possible plans to increase its size substantially. ¹²² Furthermore, ATP was approaching the due date for a mandatory report to Congress on its achievements to date, and needed additional material for the report. ¹²³

The average length of the projects in the population to be surveyed was three years. A few of the first-funded projects had reached the end of ATP funding. Nearly half of the 60 projects surveyed had completed at least 50% of their research goals, while another 40% had completed 75% or more. Yet nearly all of the companies were still in the process of carrying out research to develop their technologies. Hence, the focus of the survey was still on early effects and not on ultimate, longer-term economic impacts. NIST statisticians were consulted on sampling strategy. Their advice to ATP was to survey 100% of the projects, and to include as many participating organizations as possible. This approach was recommended in order to allow the data to be analyzed in a variety of ways while maintaining sufficient responses for statistical significance.

ATP engaged an independent contractor to carry out the second survey. Again, it was decided that the survey would be conducted by telephone interview with a single, qualified analyst, Bohne Silber, conducting all of the interviews. ¹²⁴ This second survey used a detailed questionnaire, consisting mainly of closed-ended questions in contrast to the emphasis on open-ended questions in the first survey. The contractor worked closely with ATP staff to obtain background on the program and to develop the questionnaire, which was pre-tested and revised several times before it became final. Two versions of it were created to decrease the time burden: a “long form” to be administered to all single company applicants and to joint venture participants who indicated progress in commercializing results from their ATP projects, and a “short form,” without the questions on commercialization and business, to be administered to joint venture participants who indicated no current involvement in commercialization efforts. Wherever possible, conditional questions were used that would screen out a number of subsequent questions and branch to another part of the questionnaire. For example, if the interviewer received a negative response to the question “Have you engaged in collaborative relationships?” then none of the subsequent questions on collaboration would be asked. The questionnaire in its long form contained 134 questions and took about 90 minutes to administer if all the questions were applicable. The short form took 70–75 minutes if all the questions were applicable. In most cases not all the questions were applicable.

Respondent were informed in advance that their responses were confidential and would not be revealed to ATP staff. The purpose was to encourage the company representatives, most of whose companies were still receiving funding from ATP, to be candid in their responses. A disadvantage of this approach was that ATP did not receive the data files, and had to go back to the contractor for subsequent analysis of the data.

The survey collected background information on each organization and project. It asked if any positive or negative effects on the environment, health, or safety were expected, since the detailed questions focused on direct economic effects. ¹²⁵ It asked the respondent to classify the long-run expected technical outcome of the project in terms of a new or improved product, service, or process. It also asked the respondent to classify the level of progress toward accomplishing their R&D plan. Because assessment of projects’ technical achievements was otherwise disaggregated in ATP and held in the files of individual ATP technical program managers, survey statistics provided the only consolidated measure of technical progress for most of the program’s first decade. ¹²⁶

The survey focused, in part, on the respondent’s plans to commercialize their developing technology. The inquiry attempted to identify any emergent commercial applications of the technology that were not in the original proposal. New applications are significant because it indicates that a funded technology is enabling a variety of commercial activities. The inquiry also probed the companies’ progress in bringing their technology to market in a specific application. To illustrate the level of detail, Table 5–3 gives an example of a series of commercialization questions taken from the Silber survey questionnaire. As would be expected, companies farther along in their research projects were more likely to exhibit commercial activity.

Another focus was on capturing information that would suggest the generation of knowledge spillover benefits. Table 5–4 illustrates several questions relating to that topic.

Collaborative relationships were another featured topic on the questionnaire. The intention was to discover the extent and purpose of collaboration, the role played by ATP, how well the collaborative relationships were working, and who was collaborating with whom. Table 5–5 gives examples of questions related to collaboration.

In addition to the topics of Tables 5–3 to 5–5, the survey covered employment effects, competitive standing, attraction of additional funding, and leveraging of research funds. This survey remained a primary source of performance data for ATP for two years. The survey questionnaire, which is available to the public, also provided the basis for developing ATP’s next major survey tool.

Establishing Routine Project Reporting by Electronic Survey

By the mid-1990s, ATP decided to survey all organizations participating in its funded projects on a regular basis. It decided to replace the occasional telephone survey administered by an outside contractor with an electronic survey administered by ATP staff. ATP staff further refined and substantially extended the previously used questionnaire to better track the progress of individual projects towards delivering economic benefits, and to collect data for reporting under the Government Performance and Results Act of 1993 (GPRA).

Table 5–3. Illustrative Survey Questions on Commercial Progress

Source: Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, Survey Questionnaire, Appendix A.

Table 5–4. Illustrative Survey Questions Relating to Knowledge Spillovers

QUESTION: Do you disseminate or plan to disseminate non-proprietary information—again, we’re speaking about your work that’s not confidential—about the ATP-funded technology through any of the following:

Source: Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, Survey Questionnaire, Appendix A.

Several advantages were seen to having ATP administer and manage these next surveys of project participants. ATP staff is under strong legal requirements to protect the proprietary and confidential information of applicants and award recipients and to abide by nondisclosure rules. The companies are accustomed to interacting with ATP staff, and would require less administrative time than is needed of ATP staff to help coordinate interactions between the contractor and the companies. Most important, maintaining internal control of the survey would make it easier for ATP to construct an integrated set of databases supporting the comprehensive statistical analyses of all participants in all projects—an important tool of project and program management. Internal control over project data would give ATP more flexibility in generating a variety of analytical reports on a fast turnaround basis to respond to specific stakeholders’ requests.

Table 5–5. Illustrative Survey Questions about Collaborative Partners

QUESTION: For each of the following types of collaborative relationships you have had, tell me how well overall the collaboration is working:

Source: Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, Survey Questionnaire, Appendix A.

The ATP-administered, electronic survey of project participants is known as the “Business Reporting System” (BRS). ¹²⁷ It covers all projects from 1993 forward, and pushed beyond the Silber survey in gathering data on technical and economic progress. Its coverage extends not only to the lead companies, but also to other joint venture participants, universities, and not-for-profit organizations. Until recently, the BRS was administered using customized diskettes mailed from ATP to the respondents, completed by them, and returned to ATP for downloading into the BRS database. As its first decade ended, the BRS was largely converted to a web-based system.

To help ATP establish a baseline, project participants are required to report at the outset of their project their planned application areas for the technology and their strategies for commercialization. At the end of each year, participants report on progress toward implementing their commercialization strategies, short-term economic impacts of their projects, and any changes in plans. At the conclusion of their projects, they report on their overall accomplishments. During the post-project period, they report three additional times—every other year over six years, according to the “Terms and Conditions” of their cooperative agreement. Extending the survey six years beyond project end is ambitious because the difficulty in tracking project effects tends to increase over time. Difficulties may stem from personnel changes within the firm, dimming memories about the ATP-funded part of the effort, transfer of developing technology to other parts of a company or to collaborators who may not know or care about the ATP-source of research funding, other sources of funding becoming predominant, mergers and acquisitions, and a blending of the ATP-supported technology with other technologies that blurs its ATP origins.

Over time, the emphasis in the BRS reports shifts increasingly from early indicators of progress to economic impacts. Table 5–6 summarizes the several parts of the BRS.

The resulting database is a unique management, policy, and evaluation research tool. It captures the linkage of technologies under development to applications in numerous industry sectors. It allows ATP to see major tendencies and emerging trends across its portfolio of projects. The data can support varied analyses by industry sector, technology area, geographical location, funding year, collaborative relationships, type of ATP competition, organizational size and type, and other characteristics.

Since the beginning of the program, for example, small business advocates have worried that small companies might not fare well in the program relative to larger companies. BRS data allows evaluators to compare small firms with larger firms with respect to their participation, strategies, and commercial progress. ¹²⁸

Table 5–6. Components of ATP’s Business Reporting System

Source: Powell, “The ATP’s Business Reporting System: A Tool for Economic Evaluation,” 1996.

Table 5–7 lists some of the dimensions on which Powell has compared small and large firms. She computed Z-test statistics to determine the level of significance of the statistical difference in the two groups. Findings, discussed in Chapter 9, suggest that small companies are thriving in the program.

Because of the confidential and proprietary nature of much of these data, ATP publishes results in aggregate form only.

Table 5–7. Examples of Variables for Comparing

Source: Powell, “ Business Planning and Progress of Small Firms Engaged in Technology Development Through the Advanced Technology Program ,” 1999.

Soliciting Feedback by Survey on Customer Satisfaction and Marketing Issues

In addition to providing statistics that describe program effects, surveys can be used to assess customer satisfaction and to address marketing issues. Applied to a public program such as ATP, customer satisfaction means determining how well the relationship is working between the program and the organizations with which it directly interfaces in carrying out its mission. In ATP’s case, the organizations it funds may be thought of as counterparts to a business firm’s customers.

The use of the term “customer satisfaction” in this context does not imply that the funded businesses are ATP’s ultimate customer. As a public program, U.S. citizens are ATP’s ultimate customers. But as a counterpart to business use, the funded organizations are those with whom ATP deals directly, and the success of those interactions affects the success of the program. In the case of a public program, which by definition is operating outside private markets, “marketing issues” refers to questions about how the services offered by the program become known by, and are perceived by, the public.

Soliciting early feedback from customers is particularly important when a new endeavor creates many new sets of relationships. It is preferable to find out as soon as possible how well the relationships are working, rather than simply to assume they are working well. But even though program administrators want to know how they are doing in these new relationships, they are not immune from disliking criticism and being defensive in the face of it. For this reason a third-party assessment, with full confidentiality to respondents, is essential.

ATP’s published customer satisfaction survey met these tests. It was conducted as part of the second contractor survey, and the results included as part of the larger report. ¹²⁹ Questions were asked about the resources and personnel of NIST technical support, the ATP professional staff, and all aspects of the program over which ATP has control, including the solicitation of proposals, review and evaluation of proposals, selection of award recipients, and project monitoring.

Respondents were also encouraged to give specific comments regarding their views about working with ATP. ¹³⁰

A marketing issue was also addressed by this survey. Respondents were asked how they learned about ATP. The objective was to find out which outreach efforts were most effective. Several other marketing issues have been addressed by other surveys.

After its first competition, ATP received complaints from several companies that the cost of writing a winning proposal was too high. In its first survey ATP included a question about cost of proposal preparation in order to examine this complaint. ¹³¹ Survey findings indicated that winning proposals ranged in preparation costs from a low of several thousand dollars to a high of several hundred thousand dollars. The fact that proposals with low preparation costs were able to win awards supported ATP’s contention that proposal content—not preparation cost—mattered most. At the same time, the perception of prospective applicants about preparation costs is likely an important factor in shaping their decisions on whether or not to submit proposals.

The question of proposal cost was re-surveyed nearly 10 years later by Feldman and Kelley, ¹³² with much the same result: Reported preparation costs were extremely variable across winning proposals, ranging again from several thousand dollars to several hundred thousand dollars. ¹³³

The survey by Feldman and Kelley addressed two other issues important from a program marketing perspective: (1) Do the applicants, regardless of the outcome, consider the ATP review and selection process fair; and (2) do they plan on applying to ATP again in the future. ¹³⁴ The majority, whether they received an award or not, viewed the process as fair, and the majority planned on applying to the program again.

Using Survey for Case Study and to Address Research Questions

The previous examples of the survey method focused on providing aggregate descriptive statistics for a program overall. But the survey method can also be used to collect data for an individual project case or to investigate particular research questions. This section illustrates the varied use of the survey method in evaluation research, using examples from ATP.

Surveying Joint Venture Members to Compile Case Study Data

The survey method can be used to gather data in support of other studies. In fact, in the first case study of an ATP joint venture project, Professor Albert Link of the University of North Carolina-Greensboro used a survey to collect economic data from participants. ¹³⁵ The survey targeted participants in the Printed Wiring Board (PWB) research joint venture, a five-year effort aimed at a turnaround in an industry sector in sharp decline. To establish a lower bound estimate of the economic value achieved by the joint venture, ¹³⁶ the survey examined a subset of project tasks that participants said they would have started even in the absence of ATP support thereby introducing a counterfactual element. All members of the joint venture completed the survey for each of five major research groups.

The survey had three parts. In the first part of the survey, the counterfactual part, Link asked joint venture members to quantify, by project task, a number of related metrics comparing the actual project technological state to a hypothesized technological state that would have existed at the same time in the absence of ATP’s financial support of the joint venture. From the results, he identified cost and time savings based on those research tasks the members thought they would eventually have done without ATP, and he separated out the tasks they otherwise would not have done at all.

In the second part of the survey, Link collected technology diffusion information: number of papers presented; number of conferences attended in which joint venture members talked about the project’s activities; and percentage of PWB supplier industry with which members interfaced in conjunction with the project. In the third part of the survey, Link collected information about changes in international competitiveness that members believed were linked to the PWB joint venture. He looked at changes in the companies’ share of each market segment due to their participation in the project, and changes in the United States PWB industry’s share in world markets due to accomplishments of the PWB joint venture as a whole. Survey questions regarding the competitive position of member companies are shown in Table 5–8.

Table 5–8. Competitive Position of Member Companies in the World PWB Market

As a result of my company’s involvement in the PWB program, my company’s share of each of the following segments of the PWB market has (choose one for each applicable market segment): increased; stayed the same; decreased; no opinion.

Source: Link, Advanced Technology Program; Early Stage Impacts of the Printed Wiring Board Research Joint Venture, Assessed at Project End, 1997, p. 43.

Using Survey to Explore Research Questions

The survey method may be applied to individual research questions. For ATP, a question of central importance is what difference ATP makes for the projects it funds. Or, expressed counterfactually, what would have happened had there been no ATP. These questions are fundamental to both the politics and economics of the program.

As indicated previously, establishing the fact that the larger portfolio of ATP-funded projects has had a substantial positive impact is a necessary but insufficient argument for the program’s existence. Both economists and politicians want to know how much of the impact is attributable to ATP. Multiple evaluation methods have been used to answer this question, and prominent among them was the survey method. An account of how the survey method has been used to tackle this difficult question is instructive to the field of program evaluation. ¹³⁷ It is a story that demonstrates increasing sophistication in both the questions asked and the efforts to answer them.

Early Surveys Address Counterfactual Question

In accordance with good evaluation practice, both of the early contractor surveys of ATP project participants attempted to isolate the effects attributable to ATP. The first survey ¹³⁸ simply asked participants “with what likelihood would your organization have pursued the development of your technology, without the ATP award.” If the response indicated they likely would have pursued the technology development, a follow-on question was asked: “Without the ATP award, would you have pursued it at about the same level of effort, with the same ultimate goal?” Almost all the program participants responded either that they would not have pursued their technology development projects at all without ATP or not with the same level of effort and not with the same goal. Anecdotal information was solicited of respondents about how the projects would have differed without ATP. This finding conforms to ATP’s expectations since ATP’s selection process was geared not to fund projects that were expected to proceed in the same way without ATP funding. ¹³⁹

The second survey ¹⁴⁰ probed deeper to determine the effects attributable to ATP. Table 5–9 lists some of the companies, including their number sequence in the survey as reference. A combination of open-ended and closed-ended questions was used, and similar questions were approached in several different ways. Results from both of these earlier surveys supported the conclusion that ATP made an important difference in the timing, resources, and level of risk exposure of participating companies. The effects of ATP as indicated by the responses to these counterfactual questions appeared totally consistent with its mission. But opponents continued to charge that the effects would have happened anyway. To better respond to stakeholders, particularly skeptics, ATP’s evaluation efforts continued to pursue the counterfactual question with more diligence.

BRS Survey Regularly Probes Counterfactual Question

When ATP implemented the BRS electronic survey system in 1993, the question of ATP’s impact—apart from the impact of the projects themselves—was high on the list of evaluation study topics. The BRS survey adopted many of the questions in the Silber survey and added more on timing, resource commitment, and risk acceptance. Indeed, counterfactual elements were incorporated throughout the BRS survey. The objective was to test in multiple ways whether ATP funds were leveraging private investment dollars, or substituting for them; whether ATP was encouraging companies to take on more technically challenging projects than they otherwise would; and whether it was accelerating technology development.

Table 5–9. Survey Questions Probing the Counterfactual: With and Without ATP

Source: Silber & Associates, Survey of Advanced Technology Program 1990–1992 Awardees: Company Opinion About the ATP and its Early Effects, 1996, Survey Questionnaire, Appendix A.

The results of the on-going BRS survey substantiated and extended results of the earlier surveys. It provided evidence that ATP accelerated the participants’ technology development, enabled them to take on high-risk R&D, and stimulated them to spend more of their own funds on R&D than they would otherwise have invested.

Still, opponents of the program continued to assert that ATP simply substituted public dollars for private R&D dollars and did not cause the effects found by the evaluation studies. This view held strong and steady in some quarters despite a 1996 survey by the General Accounting Office (GAO) of “near winners,” ¹⁴¹ that found about half discontinued their projects altogether when they failed to win an ATP award, while nearly all that continued did so at a reduced level of activity.

This finding was consistent with the finding of the earlier surveys that ATP was either enabling projects to start that otherwise would not have started, or accelerating or expanding the scale or scope of those that would have started—effects totally in keeping with its legislated mission. Feldman and Kelley confirmed the findings in a later survey. ¹⁴²

Focused Surveys Seek Details on ATP’s Effect

In 1996, a study was launched which used the survey method to explore the details of how and why ATP might accelerate technology development and commercialization, what this might be worth to companies, and whether there were effects on timing that extended beyond the ATP project. The survey, carried out as a doctoral dissertation, ¹⁴³ used structured telephone interview of company participants in 28 projects funded by ATP in 1991. All of the questions centered on the companies’ applied research and technology development cycle time. The purpose was to see if more light could be shed on ATP’s purported effect on technology development time.

The survey found that reducing applied research cycle time was important to the participating companies. It found that they gave concrete reasons for the importance of reducing research time, that the median time savings from ATP participation was put at 50% or three years, and that savings in research time for most of the companies translated into faster time to market. Most of the respondents were able to give either a quantitative or qualitative ballpark estimate of the value of acceleration, with a median value in the millions of dollars for every year saved.

The responses were compatible with ATP having a leveraging rather than a substitutive effect. Respondents listed five major ways participating in the program helped them reduce cycle time, one of which was ATP funding. They explained how some of these effects carried over to benefit other non-ATP technology development projects. The findings confirmed and extended previous survey findings and provided further evidence that ATP was making a difference. However, these findings remained insufficient proof for some.

A later survey focused on universities as research partners in ATP projects. ¹⁴⁴ The study used a random stratified sampling process to identify 54 companies for the survey. Category-specific survey instruments were faxed to each respondent, with multiple follow-ups by telephone to increase the response rate. The sample included joint ventures and single applicants, each with and without university involvement.

Recent Surveys Use Randomly Drawn Control Groups to Strengthen Tests of ATP’s Effect

The Feldman and Kelley survey of large random samples of winners and nonwinners from ATP’s 1998 competition, ¹⁴⁵ discussed earlier in the chapter, also investigated the question, “Does ATP funding make any difference?” ¹⁴⁶ Rather than ask winners what they would have done without the ATP award, this survey asked a randomly selected control group of non-winners one year after proposing to ATP if they had proceeded with their proposed projects, and if they had, how the scale of work compared with what had been proposed to ATP. The sample groups included 119 award winners and 122 non-winners. The investigators used independent sources to verify survey responses concerning employment, financing, and the founding date of the company, and matched survey records with data from the proposals and the firm’s prior history of applications and awards. The survey found that most non-winners did not proceed with their R&D plans and that most of those who proceeded pursued the project at a smaller scale than what they had p