Thus, for each program goal, a set of indicator variables are selected, each of them is weighted, the weighted values are summed, the process is repeated for each of the subsequent goals, and then the aggregated values for each goal are summed, and an adjustment factor is applied to convert the composite raw score to a 0-4 point scale used to assign 0 to 4 stars.

Challenges are to define appropriate program goals for the relevant evaluation period, identify what should be-and feasibly can be-the indicators of progress for each goal, and decide how much each indicator should count, i.e., what weighting factor should be applied to each indicator variable, in deriving the composite rating.

5.2 Defining Mission-Driven Goals for CPRS Development

The purpose of the CPRS is to provide a management and evaluation tool for the intermediate period after ATP funding ends and before long-term national benefits have had time to appear. Table 4 shows in three columns ATP's goals in relationship to its time horizon. It illustrates the period of focus.²⁵ In the left-hand column, the table shows the short-term period during the awarded research, and accomplish the research tasks of funded projects. This period, during which ATP funding occurs, ranges from one to five years for each project, and lasts an average of about three years across all projects. In the right-hand column, the table shows the period during which the long-term goals of the program are to be realized for successful projects. In this period-approximately 10-12 years after the project ends-ATP's ultimate goals may be stated in terms of "stimulating prosperity through innovation" and "broad-based benefits for the nation." The long-term goals are to achieve widely distributed social benefits, comprised of both private returns and spillover returns. Neither of these periods is the focus of this CPRS evaluation tool. Rather, it is the intermediate or interim period that is of central focus for the CPRS.

which the principal activity is research, and the principal short-term program goals are to foster collaboration, stimulate proposal development, accelerate the awarded research, and accomplish the research tasks of funded projects. This period, during which ATP funding occurs, ranges from one to five years for each project, lasts an average of about three years across all projects. In the right-hand column, the table shows the period during which the long-term goals of the program are to be realized for successful projects. In this period-approximately 10-12 years after the projects ends-ATP's ultimate goals may be stated in terms of "stimulating prosperity through innovation" and "broad-based benefits for the nation." The long-term goals are to achieve widely distributed social benefits, comprised of both private returns and spillover returns. Neither off these periods is the focus of this CPRS evaluation tool. Rather, it is the intermediate or interim period that is of central focus for the CPRS.

During the interim period, the program's funding is completed and work is no longer closely monitored by ATP project managers, who have newly funded projects assigned and others still underway to monitor. During the interim period, achievements are no longer adequately captured in terms of numbers of applications, awards, initial collaborative formations, research acceleration, and accomplishment of technical tasks. But, achievements cannot yet be captured in terms of national impacts.

During the interim period, assessment on the technical side shifts from evidence that individual project research tasks are being met, to external evidence that the project has created a body of significant new knowledge, and that the knowledge is being disseminated. On the economic side, in the absence of long-term impacts, assessment centers on evidence that someone is pursuing commercial applications of the new knowledge-particularly the innovators and their collaborators and licensees because they are expected to be positioned for early use of the technology.

Hence, the mission-driven goals for ATP's interim period are defined as (1) creating knowledge, (2) disseminating knowledge, and (3) commercializing results, and the indicator metrics for constructing the CPRS are selected to correspond to these interim goals. Although spillovers-critical to ATP's rationale and long-term success-are not called out directly, the characterization of interim goals and metrics is consistent with ATP's attention to spillovers- knowledge spillovers through knowledge creation and dissemination, and market spillovers through commercialization activities.

During the interim period, stakeholders expect ATP administrators to know how the body of projects that have completed their ATP funding are progressing and if they appear on track to deliver on long-term expectations. To this end, the CPRS was developed.

5.3 Data Constraints in Constructing ATP's CPRS

As related previously, ATP's need for metrics during the intermediate period had led, prior to the construction of the CPRS, to identification of a variety of output and outcome data to serve as indicator metrics. ²⁶ , ²⁷ In Geisler's words, "Core indicators and measures provide a picture of the outputs from S&T-as they evolve through the flow of the innovation process...." ²⁸

The categories of data uniformly collected by the ATP status reports for the first 50 completed projects included the following:

• Technical awards (including name of award and presenting organization)
• Business awards (including name of award and presenting organization)
• Patents filed-granted and not yet granted (including patent numbers and titles of those granted) and patent citation trees for projects with patents granted
• Publications and presentations (including only counts)
• Products and processes on the market or expected soon (including trade names of items on the market and counts of those expected)
• Collaborations (including types of collaborations-with joint venture research partners, subcontracts, university partners, licensing arrangements, and collaborations for commercial activities)
• Attraction of additional capital (including sources of capital, but not consistently the amounts)
• Change in small-company employment (large-company employment was excluded because of the difficulty of linking it to a single project; joint venture employment was excluded because of the complexity)
• Outlook for future developments (the case analysts' qualitative assessments of outlook)

A constraint in developing the CPRS was that it be formulated to use existing variables and data compiled for the first 50 completed projects.

6.0 Specifying Indicator Variables for Use in the CPRS Formulation

This section identifies the specific indicator variables selected for use in the CPRS formulation from those available, and discusses the selection decisions. Indicator variables were selected to signal progress toward the three major program goals identified as being particularly prominent during the intermediate period.

6.1 Variables to Indicate Progress Toward Knowledge Creation

Knowledge creation is an essential component in the rating system because it lies at the heart of each ATP-funded project. Each project, to be approved, must provide convincing evidence that it has "strong potential for advancing the state of the art and contributing significantly to the U.S. scientific and technical knowledge base." ²⁹ The technology must be highly innovative and the research challenging. A qualified research team with access to necessary research facilities must carry out the research. It is the creation of knowledge through ATP-funded research that fuels subsequent developments leading ultimately in successful projects to broad-based, national economic benefits. The challenge is to choose from the available indicator variables those that best indicate that a project has created significant scientific and technical knowledge.

6.1.1 Technical-Award Indicator

Receiving a technical award from a third-party organization provides a robust signal not only that new technical knowledge has been created, but also that it is of particular importance or significance. Many projects that successfully develop new scientific or technical knowledge will not be recognized with an award. Hence, award recognition for technical and scientific achievement was selected as a way to highlight strong performers in knowledge creation.

6.1.2 Patent Indicator

Patents codify the new knowledge of an invention or technology, and thus signal that new knowledge has been created. However, patents are an imperfect indicator because companies use strategies other than patents to protect their intellectual property, including secrecy and speed to market. Hence, the presence of patents indicates knowledge creation, but the absence of patents does not necessarily indicate that knowledge has not been created.

The status reports separated out patent data into patents filed and granted, patents filed and not yet granted, and all patents filed. It is questionable which version serves best as an indicator of knowledge creation. On the one hand, the granting of a patent is a more reliable signal of knowledge creation than the filing of a patent application, because it is possible an application will be turned down on grounds that it does not contain original ideas. On the other hand, the granting of patents can take years, and for a young program like ATP, many of the completed projects had patent filings that had not yet been granted only because sufficient time had not yet elapsed. For this reason, patents filed was selected as the better indicator for ATP of knowledge creation, based on the assumption that the probability of overstating patents by using patent filings was smaller than the probability of understating patents by using patents granted, because many ATP project patents were still under review at the U.S. Patent and Trademark Office. Only patents filed during or after the ATP project, and that were a result of ATP-funded research, are counted.

6.1.3 Publications and Presentations

Publishing and making presentations are hallmarks of most research and generally accompany knowledge creation; hence, this data category also deserves consideration as an indicator of knowledge creation. Although publishing and patenting may be inhibited in cases where secrecy is an important strategy for protecting the company's intellectual property, company researchers are often able to publish and present nonproprietary aspects of their research.

The available status report data for publications and presentations consisted only of counts, providing no ability to adjust for differences in quality. The data do not distinguish between a peer-reviewed paper in a leading technical journal and an unreviewed paper in trade magazines or conference proceedings. Further, the distinction between published papers and oral presentations is not consistent. For these reasons, the combined number of publications and presentations is simply taken as a rough indicator of knowledge creation, even though it is not an ideal measure.

6.1.4 Products and Processes

To compensate for the fact that some companies strive to keep their knowledge creation secret, declining to publish, present, or to patent even when they have created new knowledge, it is important to look downstream to see if products and processes result from the research project. The emergence of new and improved product and processes can help to pick up knowledge creation missed by the other indicators.

The status reports captured both new and improved products and processes in the market and those expected to be in the market soon. To be counted, it was necessary that the products and processes be well defined and the evidence convincing to the analysts that commercialization had occurred or was imminent. The status-report case analysts provided trade names and product/process descriptions. But the indicator variables do not reliably distinguish between those products and processes that represent revolutionary or breakthrough ideas and those that represent only modest contributions. Hence, the number of new and improved products and processes provides only a rough indicator of knowledge creation.

6.1.5 Summary

The average project length is approximately three years and the post-project period before the data are compiled is 3 to 5 years; this means that most projects have 6-8 years after project award to show one or more of the above indicators that knowledge has been created. The CPRS gives the project credit for knowledge creation if there are awards for new technologies created, patent filings, publications or presentations, and new products or processes on the market or expected soon after the end of the period covered.

6.2 Variables to Indicate Progress Toward Knowledge Dissemination

The dissemination of knowledge is an important pathway for generating spillovers and the broadly diffused benefits that are an ultimate goal of ATP. Even if the award recipient fails to carry the technology into the marketplace, others may take up knowledge disseminated from the project and make something of it. And, should the award recipient successfully carry the technology into the marketplace, the dissemination of knowledge will provide an additional pathway for spillover benefits, a pathway that increases the overall potential for broad-based national benefits.

The challenge is to select from available indicator variables those that can best serve as indicators of knowledge dissemination. Some of the same variables are indicative of both knowledge creation and dissemination. Differences in the weights assigned are used to take into account differences in the strength of the relationship of a given indicator variable to knowledge creation versus knowledge dissemination.

6.2.1 Patents

Patents, by codifying new knowledge, provide a means for disseminating it. As indicated in the list of data categories provided in Section 5.3, in addition to patent counts, patent trees were constructed for the projects which had patents granted. The trees show who cited the patents, and who, in turn, cited those citations, and so forth. Because they show the intensity of citing and who is citing, patent tree data offer a potentially better indicator of knowledge dissemination than patent count data. But there are several challenges. One is how to convert the complex citation data to an indicator measure, including whether to treat foreign organizations that cite the patents differently than domestic organizations. Another challenge is what to do about the substantial numbers of patent filings, not yet granted, for which there are no patent trees. Because of unresolved issues associated with using the patent citation data and the incomplete nature of the data, the prototype CPRS was formulated to use the number of patents filed as an indicator of knowledge dissemination.

6.2.2 Publications and Presentations

Publications and presentations are a primary means through which knowledge is disseminated. Furthermore, it is a means of dissemination that is easily and inexpensively accessed. As noted earlier, the data collected support only counts-not quality-of publications and presentations.

6.2.3 Products and Processes

Through inspection and reverse engineering, knowledge can be gleaned from products and processes in the market. Obtaining project information in this way, however, tends to require more effort, be more costly, and to entail a greater lag from the time of the initial research than patents and publications. Nevertheless, the existence of products and processes appears a valid indicator that knowledge may be disseminating.

6.2.4 Technical Awards

Technical awards serve to call attention to new technology and, thereby, further knowledge dissemination. Hence, a technical-awards variable is included among the dissemination indicators.

6.2.5 Collaborations

An additional indicator of knowledge dissemination is the existence of collaborative relationships. Through collaborative relationships among researchers, and between researchers and commercial partners, project knowledge is shared. Though the extent of knowledge dissemination through collaborative relationships may be constrained in terms of the size of the affected population, the effectiveness of the dissemination tends to be strong.

6.3 Variables to Indicate Progress Toward Commercialization

6.3.1 Products and Processes

Products and processes in the market or expected soon are a direct indicator of commercial progress. Market presence signals that a project has progressed to the point that economic benefits may begin to accrue. It is taken in the CPRS formulation as the principal available indicator of commercial progress.

6.3.2 Attraction of Additional Capital

Attraction of additional capital is considered a useful indicator of progress toward commercialization because it shows that additional resources are being made available for further development and commercial efforts. Attraction of capital is generally taken as a signal that the level of technical risk has been sufficiently reduced that others are willing to invest to take the technology into use.

The available collaboration data included to some extent identification of the sources of funding: innovator financing through public stock offering or retained earnings, funding by other federal agencies and by state government investment funds, and funding through collaborative commercialization agreements. In some cases, the amount of funding by source was also provided, but not consistently. Hence, there was no way to compare the resources resulting from the alternative sources and no way to know if having multiple funding sources indicated more or less resource strength than having a single funding source. For these reasons, the indicator that was adopted for the CPRS formulation was simply whether or not additional funding had been obtained for continuation of the objectives. Information on the various sources of funding, the number of funding sources, and the partial data on the amounts of funding were not included.

6.3.3 Employment Change

A potentially useful indicator of commercialization is employment gains, but linking employment changes to a particular project may be difficult or impossible. In small companies, it is more reasonable to link a particular project to company growth. In large companies a host of other factors typically influence employment, making employment change an unreliable indicator of a project's commercial progress. In the case of joint ventures, tracking employment changes associated with a given project along task lines tends to be complex.

For these reasons, employment change data was collected by the status reports only for small, single-applicant companies. For these small companies, employment data were recorded at the project start and after the project ended by case-study analysts, and the percentage change was recorded. The small-company employment change is included in the CPRS formulation as an indicator of commercialization progress for these companies. Because most of the participants in the first 50 completed ATP projects were small companies, this indicator was available for most of the projects in the first application of the CPRS. For the large companies and joint ventures, the employment change indicator was assigned a default value.

6.3.4 Business Awards

In addition to awards for scientific and technical achievements, awards are given by third-party organizations to businesses that are demonstrating unusual business acumen. These awards are often made to small companies that are growing at a rapid rate. For small emerging businesses with a single technology focus-which describes many participants among the first 50 completed ATP projects-business awards appeared closely linked to the commercialization of the ATP-funded technologies. Hence, the CPRS was formulated to include a business-award variable for use as an indicator of commercial progress.

6.3.5 Outlook

For each of the first 50 completed ATP projects, the case-study analyst provided a qualitative assessment of future prospects. This allowed the analyst to bring in information beyond that revealed by the other indicator data. For example, the analyst might have found outputs and outcomes suggesting a relatively robust project, but also discovered that an alternative approach was expected soon to displace the project's technology, such that the outlook for long-term benefits was pessimistic. Or, there might be little in the data to suggest a robust project, but the case analyst might uncover a new development in the works that would give cause for optimism.

To facilitate translating the outlook descriptions into an indicator of commercial progress, the projects were divided into three groups with respect to outlook. Group 1 included those projects whose outlook the case-study analyst described as highly promising, or excellent, or on track. Group 2 included three subgroups: those whose outlook was described as neither particularly strong nor weak; those whose outlook was described as promising but with serious reservations or qualifications added; and those whose outlook was uncertain. Group 3 included those projects whose outlook was portrayed in clearly pessimistic terms. The CPRS was formulated to include a numerical outlook variable, reflecting the outlook group to which a project is assigned.

7.0 Applying Weighting Algorithms to the Selected Indicator Variables and Calculating Scores

Weights were assigned to each of the selected indicator variables to determine how they figure in the composite rating. As was the case with the other composite scoring systems reviewed in Section 7, expert judgment was used to determine the weights for the CPRS.

The range of values observed for each of the variables in the database compiled for the first 50 completed ATP projects influenced the specification of weights. An objective was to dampen the effect of outlier values. Applying the assigned weights to each indicator variable produced components of the raw score. Summing the components of the raw scores resulted in the composite raw score that was then converted to a star rating.

7.1 Knowledge Creation Scoring

Table 5 summarizes the weighting of indicator variables selected to indicate progress toward knowledge creation. Column 1 lists the variables in declining order of their assumed importance as indicators of knowledge creation. Column 2 shows the range of values for each variable observed for the first 50 completed ATP projects. Column 3 shows the weighting algorithm for each variable. Column 4 gives the range of raw scores calculated for the first 50 completed projects by applying the weighting algorithm to each indicator variable. The bottom row of the table shows the range of aggregated raw scores for knowledge creation.

7.1.1 Weighting the Technical-Award Indicator

Technical awards among the first 50 completed ATP projects ranged from zero to four in number. That is, some projects received as many as four awards from different organizations. Technical awards are assumed to serve as the best indicator that significant knowledge was created. The awarding of multiple awards by different organizations seemed largely independent of one another, and the decision was made to assign equal weights to each additional award. The raw score for technical awards is calculated by multiplying 1, the weight, times N, the number of science and technical awards received. The raw scores for awards ranged from 0 to 4 for the first 50 completed ATP projects.

7.1.2 Weighting the Patent Indicator

Patent filings ranged in number from 0 to 26 per project, with the range reflecting patenting strategy as well as the amount of knowledge created. That is, one project may file a single patent to capture its knowledge creation, while another may file many patents to capture a comparable or different amount of knowledge. The weighting algorithm values the first patent at half that of a technical award, and additional patents at a sharply decreasing rate. The raw score for patent filings is calculated as 0.5 times the square root of the number of patent filings. The raw score for patent filings ranged from 0 to 2.5 for the first 50 completed ATP projects.

7.1.3 Weighting the Publication/Presentation Indicator

Publications and presentations ranged in number from 0 to 214, with one project having far more than the rest. The weighting algorithm values the first publication or presentation the same as the first patent, and additional publications and presentations at an even more sharply decreasing rate. The raw score is calculated as 0.5 times the fourth root of the number of publications and presentations. The raw scores ranged from 0 to 1.9 for the first 50 completed ATP projects.

7.1.4 Weighting the Product/Process Indicator

Products and processes in the market or expected soon ranged in number from 0 to 5. Some projects had more than one product, some a combination of product and process. Where there were multiple products or processes for a project, it is assumed they all serve to indicate the same underlying body of knowledge creation. Hence, having multiple products and processes does not increase the raw score. The weighting algorithm is binary: 0 if there are no products or processes; 0.5 if there are. This assigns the same value to having any products and processes as to having a single patent or publication. The raw scores ranged from 0 to 0.5 for the first 50 completed ATP projects.

7.1.5 Aggregate Raw Scoring of Progress Toward Knowledge Creation

The aggregate score for each project is computed by summing across the scores for the four indicators. The aggregate raw scores for knowledge creation for the first 50 completed ATP projects ranged from 0 to 8.9.

7.1.6 Sensitivity Testing of Knowledge Creation Scores

Table 6 summarizes the results of sensitivity testing of the aggregated raw score for knowledge creation to changes in the values of each of the indicator variables. 30 An inspection of the tabular values reveals the marginal contribution of each indicator to the aggregate raw score for knowledge creation. For example, rows 1-4 show that the first technical award increases the raw score by 1.0; the first patent filing, 0.5; the first publication or presentation, 0.5; the first product or process, 0.5. If a project had one of each of the indicator variables, the raw score would be 2.5. Comparing rows 1-4 with rows 6-9 shows the marginal contribution of the second unit of each indicator. Receiving a second technical award adds 1.0 to the raw score; filing for a second patent adds 0.2 to the raw score; publishing or presenting a second paper adds 0.1 to the raw score; adding another product or process in the market adds 0.0 to the raw score. The weighting scheme moderates the effect of unusually high rates of publishing and patenting for one of the projects in the sample, as can be seen by comparing rows 17-19 with rows 2 and 3. Having 1 publication yields a score of 0.5, whereas having 200 yields a score of 1.9. Having 1 patent filing yields a score of 0.5, whereas having 20 patents yields a score of 2.2. Row 19 shows the effect of inserting values toward the upper end of the observed ranges for each of the indicator variables at once.

7.2 Knowledge Dissemination Scoring

Table 7 summarizes the application of weights to the variables selected to indicate knowledge dissemination. Column 1 lists the variables in declining order of their relative weights. Column 2 shows the range of values for each selected variable observed in the database. Column 3 shows the weighting algorithms. Column 4 gives the range of raw scores for each variable calculated by applying the weighting algorithm to the corresponding indicator variable. The bottom row of the table shows the range of aggregated raw scores for knowledge dissemination across the first 50 completed ATP projects.

7.2.1 Weighting the Publication/Presentation Indicator

A step function was used in weighting the dissemination value of publications and presentations. A single publication or presentation will result in a weight of 1. Additional units up to 10 add to the score at a sharply declining rate; and units in excess of 10 add at an even slower rate. The assumption is that when a project produces multiple publications and presentations they are closely related, and each subsequent publication or presentation does not convey as much information as the previous one. 31 The approach avoids having one project's extremely large number of publications/presentations overwhelm all other variables in the aggregate raw score. The starting range of 0 to 214 translates to a range of weighted raw scores of 0 to 4.6 after the application of the weighting algorithm.

7.2.2 Weighting the Patent Indicator

The step-function weighting algorithm for patent filings is identical to that for publications and presentations, based on a similar rationale that as the number of patents in a project increases, the contribution to knowledge dissemination of additional units declines. The range of weighted raw scores is 0 to 3.6.

7.2.3 Weighting the Collaboration Indicator

Three forms of collaborative relationships are tracked, namely (1) collaborations between award recipients and university researchers, (2) R&D collaborations among award recipients and other firms, state and federal laboratories, and other non-university organizations, and (3) collaborative ties between award recipients and other firms for technology commercialization. The weighting algorithm assigns a weight of 1 to each of these forms of collaborative relationship found in a project. The rationale is that each type of collaboration provides a different pathway of knowledge flows. Among projects in the sample of 50, some had no collaborative relationships; some had one, two, or three of the three forms listed. Hence, the range of weighted raw scores is 0 to 3.

7.2.4 Weighting the Product/Process Indicator

The weighting algorithm for products and processes effectively treats products and processes as half as important as patents and publications/presentations as knowledge disseminators. The rationale for the lower weight is the additional effort and difficulty in extracting knowledge by inspection and reverse engineering of product and processes. The weighted raw score ranges from 0 to 1.1, depending on the number of products and processes.

7.2.5 Weighting the Technical Award Indicator

The weighting algorithm for receipt of technical awards effectively treats technical awards as one-fourth as important as patents and publications/presentations. The weighting algorithm reflects the fact that technical awards raise awareness of a new technology, and hence, further disseminate knowledge, but the award does not itself typically convey much detailed knowledge. Since multiple awards may call greater attention to the new technology, the weight increases slightly with increasing numbers of awards. The weighted raw score ranges from 0 to 0.5, based on awards that range from 0 to 4.

7.2.6 Aggregate Raw Scoring of Progress Toward Knowledge Dissemination

The aggregated weighted raw score for knowledge dissemination ranges from 0.7 to 12.8. The lower end of the range is positive, despite the fact that the lower end of the ranges for each of the component indicator variables is 0, and, further, that the low-end of the range for the aggregate weighted raw score for knowledge creation is 0. This may seem a contradiction, but is not for two reasons. (1) The knowledge dissemination score contains collaboration as an indicator of progress, and one of the joint-venture projects in the sample exhibited none of the measured indicators except collaborative activity. (2) Although one or more projects had 0 values for each of the other variables, no project had 0 values for all of the variables.

7.2.7 Sensitivity Testing of Knowledge Dissemination Scores

Table 8 summarizes sensitivity testing of the aggregated raw score for knowledge dissemination to changes in the values of indicator variables. Rows 1 through 5 show the value of the first unit of each variable, with a single publication or presentation resulting in a score of 1.0; a single patent, 1.0; a single type of collaboration, 1.0; a single product or process, 0.5; and a single technical award, 0.3. Row 6 shows that if one of each of the indicators were obtained, a raw score of 3.8 results.

Comparing rows 7-12 with 1-6 shows the marginal effect of adding the second unit of each variable. The second publication or presentation adds 0.4 to the score. The second patent filed adds 0.4 to the score. The second type of collaboration adds 1.0 to the score. The second product or process adds 0.2, and the second technical award adds 0.1. Having two versus one of each of the indicators results in a score of 5.9, compared with 3.8.

The small differences revealed by comparing rows 13, 14, and 15 reflect the step function used in the weighting algorithm for publications (and also patents). Increasing the number of publications and presentations from 10 to 11 increases the raw score by only 0.1. Increasing the number from 11 to 200 increases the raw score by only 1.2. Rows 15-19 show the resulting raw score if values near the upper end of the range are used in turn for each of the indicator variables. Row 20 shows the resulting raw score of 12.6 from using values near the upper end of the range for all of the indicator variables at once.

7.3 Commercialization Progress Scoring

Table 9 summarizes the application of weights to the variables selected to indicate commercialization progress. Column 1 lists the four variables used to indicate progress toward commercialization. Each of the variables used has a potentially substantial impact in the scoring. Column 2 shows the range of values observed for each indicator variable in the database of the first 50 completed ATP projects. Column 3 presents the weighting algorithms applied to the indicator variables to calculate the raw scores, shown in column 4. The bottom row shows the range of aggregated raw scores for commercialization.

7.3.1 Weighting the Product/Process Indicator

Products and processes in the market now or expected soon are given a relatively heavy weight, because it indicates the likelihood that a project has progressed to a stage in which economic benefits may result. A primary interest is that there be a product or process. The weighting algorithm reflects the relatively large importance attached to this indicator, with the first product/process receiving a score of 4.25. Additional products/processes contribute less. The raw score ranges from 0 to 5.8.

7.3.2 Weighting the Capital Attraction Indicator

Capital attraction indicates that additional resources are being made available for further development and commercial efforts. Due to data difficulties discussed earlier, a single-value weight is assigned to a project if it has attracted funding from one or more of the sources identified, regardless of amount. The indictor variable is assigned a weight of 0 if a project has attracted no additional funding, and a weight of 3 if it has attracted funding from any of the identified sources.

7.3.3 Weighting the Employment Gains Indicator

As indicated earlier, employment data were compiled only for the small, single-applicant companies. Because most of these companies were very small at project start, the percentage increases for a number of the projects tended to be very large. Growth rates of 50% were routine, and rates approached 2000% for several projects. For this reason, the weighting strategy for small companies is to treat rates of employment change up to 50% as the norm, assigning a weight of 0 for rates of change of 50% or less. If employment increased by more than 50%, a weight of 2.5 times the fourth root of the gain in excess of 50% is assigned, providing a relatively large weight, while preventing the extremely large gains in several cases from over­whelming other indicators. In the case of bankruptcy-either of a small company, single applicant, or a leader of a joint venture-a negative weight of -6 is assigned to signal that there is a serious impediment to commercial progress through the direct path of the innovating companies. In cases other than bankruptcy, a default weight of 1.5 is assigned to projects for which employment data were not collected (i.e., large companies and joint ventures). Applying the weighting algorithm resulted in raw scores ranging from -6 to 5.2.

7.3.4 Weighting the Business Awards Indicator

The weighting algorithm for business awards is to assign 0 if there are no awards and 3.25 for one award. Additional awards add 0.25 each to the score. The weighted raw scores range from 0 to 3.8.

7.3.5 Weighting the Outlook Indicator

As indicated earlier, the subjective information in the cases was used to group the projects into three groups by outlook. The weighting strategy assigns a raw score of +4 to those in group 1, whose outlook is strong; 0 to those in group 2, whose outlook is neither clearly strong nor poor; and -4 to those in group 3, whose outlook is poor.

7.3.6 Aggregate Raw Scoring of Progress Toward Commercialization

The aggregated weighted raw scores for commercialization range from -10 to 21.7. The range for this score is much wider than for knowledge creation and dissemination scoring, and may be negative if the company goes out of business, or if the outlook is poor, or if both occur.

Scoring at the low end of the range are projects conducted by companies that went bankrupt and for which the outlook revealed no active alternative champion of the technology to take it forward. At the high end were projects conducted by award-winning businesses-particularly those that are fast growing-with commercialized products or processes based on the technology, available resources to continue development and commercialization of the technology, and an outlook for continued robust progress.

Table 10 summarizes the results of testing the sensitivity of the aggregate raw score for commercialization to changes in the value of each of the indicators, and to several changes in com­bination. The use of certain combinations of values here and in the previous sensitivity testing is not to imply that they are expected to occur in the combination shown, but rather the intention is to test the sensitivity of results to extreme values.

For the purpose of clearer exposition, two of the variables, employment gains and out­look, are entered in descriptive terms rather than in terms of the numerical value into which the description is translated. The model includes four possible conditions of employment gain, only one of which allows the score to change as a function of the amount of increase in the gain. It contains three possible outlook states.

The sensitivity testing begins with the condition of employment gain. Rows 1-3 show the effect of holding the outlook constant at Group 2 (neither clearly positive or negative and having a 0 effect on the raw score), holding other variables except employment gains constant at 0, and changing only the condition of employment gains. A joint venture or large company for which meaningful employment data were not collected receives a default score of 1.5. A small company, drawn from a group in which up to 50% employment gains were commonplace, receives a score of 0. Beyond a 50% employment gain, the score is positive and increasing at a decreasing rate, with a 100% gain receiving a score of 2.1, and a 200% gain, a score of 2.8.

Comparing rows 1 and 5 reveals the effect of changing only the outlook from Group 2 (neither clearly positive nor negative) to Group 1 (positive). The raw score increases by 4.0, showing a relatively large impact on scoring of the outlook.

Comparing rows 6-8 with row 1 shows the contribution to the raw score of the first unit of each of the other indicator variables. Commercializing a product or process adds 4.3 to the raw score; attracting additional capital adds 3.0; receiving a business award adds 3.3. Row 9 shows a raw score of 16 resulting from the coincidence of a joint venture or large company having one unit of each indicator variable, combined with a positive outlook.

Comparing row 10 with row 6 reveals the effect of commercializing a second product or process to be 0.5. Comparing row 11 with row 8 reveals the effect of a second business award to be 0.2. Because capital attraction is modeled as either yes or no, it is not tested for other possible values.

Comparing rows 12 and 13 with row 3 isolates the effect of changing only the outlook. There is an 8 point drop in the raw score as the outlook changes from positive to negative.