NIST GCR 02-834
Benefits and Costs of ATP Investments in Component-Based Software

Central to using case studies to gather data on the economic impact of ATP's support of CBSD projects was the need to ensure that the process used to select the projects for inclusion was rule-based rather than outcome-based.  Establishing selection rules that are strictly followed during sample selection increases the external validity of the study (Yin, 1984 ).

Eight case studies were planned from the beginning.  One approach to selecting the eight case study candidates would be a random selection from the 24 funded projects.  Results from this approach would be the most statistically generalizable; however, by engaging in targeted sampling from the 24 projects more information could be extracted from the case studies.

Specifically, several projects that we knew a priori would not be fruitful could be eliminated from the set of 24.  We used several screening steps to narrow the list of projects, thus improving the selection of the final case studies. 

• The first screening step in selection of projects for the case studies was to eliminate all projects that were not technically successful.  Four projects were eliminated from consideration because they failed to achieve technical completion based on information provided to ATP.  It might seem that by doing this we would be placing a positive bias in our results.  However, this is not the case.  Projects that were failures are assumed to have a zero social benefit.  The estimates for the entire focused program are made more conservative by analyzing the benefits and costs of the more successful projects in the case studies while also including the costs of the failures in the portfolio analysis. 
• A second screen was used to eliminate projects that have not yielded commercial products or services.  According to the evaluation methodology detailed in the text, a project does not yield tangible economic benefits unless the technology is commercialized.  Within the CBSD focused program, two of the firms that achieved technical success elected not to commercialize their technology, and are thus effectively eliminated from the case studies.
• Two additional projects from the 1997 funding round were just completing their ATP-funded R&D phase as of December 2000, and, as such, had not reached commercialization.  These projects are also eliminated from the case-study analysis because estimating their impact would rely solely on forecasts of market size and market potential rather than on observed data.

Table A-1 summarizes the screens and the number of projects dropped during each step.

After eliminating the eight projects listed in Table A-1, we were left with a potential list of 16 projects from which to select the eight individual benefit-cost case studies.

To take advantage of the heterogeneity across projects, we wished to make selections that capture this variability.  Three major types of variations exist across the projects:  year of funding, project type, and company size.  To maximize the external validity of the study, we wished to select projects from all three rounds of funding (1994, 1995, and 1997) and all types of projects and company sizes represented in the program (joint ventures, single-company projects, projects involving start-up, and other small, medium, and large companies). 

From the list of 16 available projects, only one was led by a medium-sized company (Intermetrics), one was a joint venture (led by Commerce One, a small company at the time of the ATP award) and one was led by a large company (Xerox); as a result, all three were included in the case study group.  This selection left five projects to select from among the two classes of small firms, most of which were in the 1994 and 1995 funding rounds.  We chose to select three start-ups and two other small firms to study, in line with their representation in the total program.

To balance the rest of the case study panel, then, three decisions had to be made.  First, we wanted to choose two of the four 1994 start-ups with commercial products, which included SciComp, Aesthetic Solutions, APT, and Continuum.  Secondly, we opted to select two of the four small firms funded in 1995, which were Analogy, Reliable Software Technologies, Tom Sawyer Software, and Real-Time Innovations.  Finally, one of three 1995 start-ups would be appropriate, chosen from the group that included Extempo Systems, Hybrithms, and Semantic Design.  RTI made the final selections based on conversations with ATP about which companies would be the most willing to share information regarding their product.  Table A-2 presents the eight projects selected for inclusion in the case study analysis.

The final set of selected projects meets objectives of representation and heterogeneity, as is illustrated by the profiles presented in Tables A-3 and A-4.  These tables show the number of projects selected out of the total number of projects in that category, and those that have reached commercialization.

Table A-2.  Eight Projects Selected

^aFunded as a joint venture led by start-up firm CNgroup (later renamed VEO Systems), with partners CommerceNet, Business Bots, and Tesserae Information Systems, the last two also start-ups.  CommerceNet is an industry trade group.  CommerceOne later acquired VEO Systems, inheriting leadership of the ATP project.

^bLater merged into Averstar, now part of Hewlett-Packard.

Source:  ATP project data, including company close-out reports.

Table A-3.  Number of Projects Selected from Each Funding Year Group

Table A-4.  Number of Projects Selected by Company Size or Project Type

^a The selected JV was led by a start-up firm.  Of the two JVs not selected, one was led by a small firm, the other by a large corporation.  Neither of these JVs completed their technical work.

Return to Table of Contents or go to next Chapter.

Date created: December 3, 2002
Last updated: August 2, 2005

ATP website comments: webmaster-atp@nist.gov  / Technical ATP inquiries: InfoCoord.ATP@nist.gov.

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / ExpectMore.gov (performance of federal programs)