Thus, for each program goal, a set of indicator variables are selected, each of them is weighted, the weighted values are summed, the process is repeated for each of the subsequent goals, and then the aggregated values for each goal are summed, and an adjustment factor is applied to convert the composite raw score to a 0-4 point scale used to assign 0 to 4 stars.

Challenges are to define appropriate program goals for the relevant evaluation period, identify what should be-and feasibly can be-the indicators of progress for each goal, and decide how much each indicator should count, i.e., what weighting factor should be applied to each indicator variable, in deriving the composite rating.

5.2 Defining Mission-Driven Goals for CPRS Development

The purpose of the CPRS is to provide a management and evaluation tool for the intermediate period after ATP funding ends and before long-term national benefits have had time to appear. Table 4 shows in three columns ATP's goals in relationship to its time horizon. It illustrates the period of focus.²⁵ In the left-hand column, the table shows the short-term period during the awarded research, and accomplish the research tasks of funded projects. This period, during which ATP funding occurs, ranges from one to five years for each project, and lasts an average of about three years across all projects. In the right-hand column, the table shows the period during which the long-term goals of the program are to be realized for successful projects. In this period-approximately 10-12 years after the project ends-ATP's ultimate goals may be stated in terms of "stimulating prosperity through innovation" and "broad-based benefits for the nation." The long-term goals are to achieve widely distributed social benefits, comprised of both private returns and spillover returns. Neither of these periods is the focus of this CPRS evaluation tool. Rather, it is the intermediate or interim period that is of central focus for the CPRS.

which the principal activity is research, and the principal short-term program goals are to foster collaboration, stimulate proposal development, accelerate the awarded research, and accomplish the research tasks of funded projects. This period, during which ATP funding occurs, ranges from one to five years for each project, lasts an average of about three years across all projects. In the right-hand column, the table shows the period during which the long-term goals of the program are to be realized for successful projects. In this period-approximately 10-12 years after the projects ends-ATP's ultimate goals may be stated in terms of "stimulating prosperity through innovation" and "broad-based benefits for the nation." The long-term goals are to achieve widely distributed social benefits, comprised of both private returns and spillover returns. Neither off these periods is the focus of this CPRS evaluation tool. Rather, it is the intermediate or interim period that is of central focus for the CPRS.

During the interim period, the program's funding is completed and work is no longer closely monitored by ATP project managers, who have newly funded projects assigned and others still underway to monitor. During the interim period, achievements are no longer adequately captured in terms of numbers of applications, awards, initial collaborative formations, research acceleration, and accomplishment of technical tasks. But, achievements cannot yet be captured in terms of national impacts.

During the interim period, assessment on the technical side shifts from evidence that individual project research tasks are being met, to external evidence that the project has created a body of significant new knowledge, and that the knowledge is being disseminated. On the economic side, in the absence of long-term impacts, assessment centers on evidence that someone is pursuing commercial applications of the new knowledge-particularly the innovators and their collaborators and licensees because they are expected to be positioned for early use of the technology.

Hence, the mission-driven goals for ATP's interim period are defined as (1) creating knowledge, (2) disseminating knowledge, and (3) commercializing results, and the indicator metrics for constructing the CPRS are selected to correspond to these interim goals. Although spillovers-critical to ATP's rationale and long-term success-are not called out directly, the characterization of interim goals and metrics is consistent with ATP's attention to spillovers- knowledge spillovers through knowledge creation and dissemination, and market spillovers through commercialization activities.

During the interim period, stakeholders expect ATP administrators to know how the body of projects that have completed their ATP funding are progressing and if they appear on track to deliver on long-term expectations. To this end, the CPRS was developed.

5.3 Data Constraints in Constructing ATP's CPRS

As related previously, ATP's need for metrics during the intermediate period had led, prior to the construction of the CPRS, to identification of a variety of output and outcome data to serve as indicator metrics. ²⁶ , ²⁷ In Geisler's words, "Core indicators and measures provide a picture of the outputs from S&T-as they evolve through the flow of the innovation process...." ²⁸

The categories of data uniformly collected by the ATP status reports for the first 50 completed projects included the following:

• Technical awards (including name of award and presenting organization)
• Business awards (including name of award and presenting organization)
• Patents filed-granted and not yet granted (including patent numbers and titles of those granted) and patent citation trees for projects with patents granted
• Publications and presentations (including only counts)
• Products and processes on the market or expected soon (including trade names of items on the market and counts of those expected)
• Collaborations (including types of collaborations-with joint venture research partners, subcontracts, university partners, licensing arrangements, and collaborations for commercial activities)
• Attraction of additional capital (including sources of capital, but not consistently the amounts)
• Change in small-company employment (large-company employment was excluded because of the difficulty of linking it to a single project; joint venture employment was excluded because of the complexity)
• Outlook for future developments (the case analysts' qualitative assessments of outlook)

A constraint in developing the CPRS was that it be formulated to use existing variables and data compiled for the first 50 completed projects.

6.0 Specifying Indicator Variables for Use in the CPRS Formulation

This section identifies the specific indicator variables selected for use in the CPRS formulation from those available, and discusses the selection decisions. Indicator variables were selected to signal progress toward the three major program goals identified as being particularly prominent during the intermediate period.

6.1 Variables to Indicate Progress Toward Knowledge Creation

Knowledge creation is an essential component in the rating system because it lies at the heart of each ATP-funded project. Each project, to be approved, must provide convincing evidence that it has "strong potential for advancing the state of the art and contributing significantly to the U.S. scientific and technical knowledge base." ²⁹ The technology must be highly innovative and the research challenging. A qualified research team with access to necessary research facilities must carry out the research. It is the creation of knowledge through ATP-funded research that fuels subsequent developments leading ultimately in successful projects to broad-based, national economic benefits. The challenge is to choose from the available indicator variables those that best indicate that a project has created significant scientific and technical knowledge.

6.1.1 Technical-Award Indicator

Receiving a technical award from a third-party organization provides a robust signal not only that new technical knowledge has been created, but also that it is of particular importance or significance. Many projects that successfully develop new scientific or technical knowledge will not be recognized with an award. Hence, award recognition for technical and scientific achievement was selected as a way to highlight strong performers in knowledge creation.

6.1.2 Patent Indicator

Patents codify the new knowledge of an invention or technology, and thus signal that new knowledge has been created. However, patents are an imperfect indicator because companies use strategies other than patents to protect their intellectual property, including secrecy and speed to market. Hence, the presence of patents indicates knowledge creation, but the absence of patents does not necessarily indicate that knowledge has not been created.

The status reports separated out patent data into patents filed and granted, patents filed and not yet granted, and all patents filed. It is questionable which version serves best as an indicator of knowledge creation. On the one hand, the granting of a patent is a more reliable signal of knowledge creation than the filing of a patent application, because it is possible an application will be turned down on grounds that it does not contain original ideas. On the other hand, the granting of patents can take years, and for a young program like ATP, many of the completed projects had patent filings that had not yet been granted only because sufficient time had not yet elapsed. For this reason, patents filed was selected as the better indicator for ATP of knowledge creation, based on the assumption that the probability of overstating patents by using patent filings was smaller than the probability of understating patents by using patents granted, because many ATP project patents were still under review at the U.S. Patent and Trademark Office. Only patents filed during or after the ATP project, and that were a result of ATP-funded research, are counted.

6.1.3 Publications and Presentations

Publishing and making presentations are hallmarks of most research and generally accompany knowledge creation; hence, this data category also deserves consideration as an indicator of knowledge creation. Although publishing and patenting may be inhibited in cases where secrecy is an important strategy for protecting the company's intellectual property, company researchers are often able to publish and present nonproprietary aspects of their research.

The available status report data for publications and presentations consisted only of counts, providing no ability to adjust for differences in quality. The data do not distinguish between a peer-reviewed paper in a leading technical journal and an unreviewed paper in trade magazines or conference proceedings. Further, the distinction between published papers and oral presentations is not consistent. For these reasons, the combined number of publications and presentations is simply taken as a rough indicator of knowledge creation, even though it is not an ideal measure.

6.1.4 Products and Processes

To compensate for the fact that some companies strive to keep their knowledge creation secret, declining to publish, present, or to patent even when they have created new knowledge, it is important to look downstream to see if products and processes result from the research project. The emergence of new and improved product and processes can help to pick up knowledge creation missed by the other indicators.

The status reports captured both new and improved products and processes in the market and those expected to be in the market soon. To be counted, it was necessary that the products and processes be well defined and the evidence convincing to the analysts that commercialization had occurred or was imminent. The status-report case analysts provided trade names and product/process descriptions. But the indicator variables do not reliably distinguish between those products and processes that represent revolutionary or breakthrough ideas and those that represent only modest contributions. Hence, the number of new and improved products and processes provides only a rough indicator of knowledge creation.

6.1.5 Summary

The average project length is approximately three years and the post-project period before the data are compiled is 3 to 5 years; this means that most projects have 6-8 years after project award to show one or more of the above indicators that knowledge has been created. The CPRS gives the project credit for knowledge creation if there are awards for new technologies created, patent filings, publications or presentations, and new products or processes on the market or expected soon after the end of the period covered.

6.2 Variables to Indicate Progress Toward Knowledge Dissemination

The dissemination of knowledge is an important pathway for generating spillovers and the broadly diffused benefits that are an ultimate goal of ATP. Even if the award recipient fails to carry the technology into the marketplace, others may take up knowledge disseminated from the project and make something of it. And, should the award recipient successfully carry the technology into the marketplace, the dissemination of knowledge will provide an additional pathway for spillover benefits, a pathway that increases the overall potential for broad-based national benefits.

The challenge is to select from available indicator variables those that can best serve as indicators of knowledge dissemination. Some of the same variables are indicative of both knowledge creation and dissemination. Differences in the weights assigned are used to take into account differences in the strength of the relationship of a given indicator variable to knowledge creation versus knowledge dissemination.

6.2.1 Patents

Patents, by codifying new knowledge, provide a means for disseminating it. As indicated in the list of data categories provided in Section 5.3, in addition to patent counts, patent trees were constructed for the projects which had patents granted. The trees show who cited the patents, and who, in turn, cited those citations, and so forth. Because they show the intensity of citing and who is citing, patent tree data offer a potentially better indicator of knowledge dissemination than patent count data. But there are several challenges. One is how to convert the complex citation data to an indicator measure, including whether to treat foreign organizations that cite the patents differently than domestic organizations. Another challenge is what to do about the substantial numbers of patent filings, not yet granted, for which there are no patent trees. Because of unresolved issues associated with using the patent citation data and the incomplete nature of the data, the prototype CPRS was formulated to use the number of patents filed as an indicator of knowledge dissemination.

6.2.2 Publications and Presentations

Publications and presentations are a primary means through which knowledge is disseminated. Furthermore, it is a means of dissemination that is easily and inexpensively accessed. As noted earlier, the data collected support only counts-not quality-of publications and presentations.

6.2.3 Products and Processes

Through inspection and reverse engineering, knowledge can be gleaned from products and processes in the market. Obtaining project information in this way, however, tends to require more effort, be more costly, and to entail a greater lag from the time of the initial research than patents and publications. Nevertheless, the existence of products and processes appears a valid indicator that knowledge may be disseminating.

6.2.4 Technical Awards

Technical awards serve to call attention to new technology and, thereby, further knowledge dissemination. Hence, a technical-awards variable is included among the dissemination indicators.

6.2.5 Collaborations

An additional indicator of knowledge dissemination is the existence of collaborative relationships. Through collaborative relationships among researchers, and between researchers and commercial partners, project knowledge is shared. Though the extent of knowledge dissemination through collaborative relationships may be constrained in terms of the size of the affected population, the effectiveness of the dissemination tends to be strong.

6.3 Variables to Indicate Progress Toward Commercialization

6.3.1 Products and Processes

Products and processes in the market or expected soon are a direct indicator of commercial progress. Market presence signals that a project has progressed to the point that economic benefits may begin to accrue. It is taken in the CPRS formulation as the principal available indicator of commercial progress.

6.3.2 Attraction of Additional Capital

Attraction of additional capital is considered a useful indicator of progress toward commercialization because it shows that additional resources are being made available for further development and commercial efforts. Attraction of capital is generally taken as a signal that the level of technical risk has been sufficiently reduced that others are willing to invest to take the technology into use.

The available collaboration data included to some extent identification of the sources of funding: innovator financing through public stock offering or retained earnings, funding by other federal agencies and by state government investment funds, and funding through collaborative commercialization agreements. In some cases, the amount of funding by source was also provided, but not consistently. Hence, there was no way to compare the resources resulting from the alternative sources and no way to know if having multiple funding sources indicated more or less resource strength than having a single funding source. For these reasons, the indicator that was adopted for the CPRS formulation was simply whether or not additional funding had been obtained for continuation of the objectives. Information on the various sources of funding, the number of funding sources, and the partial data on the amounts of funding were not included.

6.3.3 Employment Change

A potentially useful indicator of commercialization is employment gains, but linking employment changes to a particular project may be difficult or impossible. In small companies, it is more reasonable to link a particular project to company growth. In large companies a host of other factors typically influence employment, making employment change an unreliable indicator of a project's commercial progress. In the case of joint ventures, tracking employment changes associated with a given project along task lines tends to be complex.

For these reasons, employment change data was collected by the status reports only for small, single-applicant companies. For these small companies, employment data were recorded at the project start and after the project ended by case-study analysts, and the percentage change was recorded. The small-company employment change is included in the CPRS formulation as an indicator of commercialization progress for these companies. Because most of the participants in the first 50 completed ATP projects were small companies, this indicator was available for most of the projects in the first application of the CPRS. For the large companies and joint ventures, the employment change indicator was assigned a default value.

6.3.4 Business Awards

In addition to awards for scientific and technical achievements, awards are given by third-party organizations to businesses that are demonstrating unusual business acumen. These awards are often made to small companies that are growing at a rapid rate. For small emerging businesses with a single technology focus-which describes many participants among the first 50 completed ATP projects-business awards appeared closely linked to the commercialization of the ATP-funded technologies. Hence, the CPRS was formulated to include a business-award variable for use as an indicator of commercial progress.

6.3.5 Outlook

For each of the first 50 completed ATP projects, the case-study analyst provided a qualitative assessment of future prospects. This allowed the analyst to bring in information beyond that revealed by the other indicator data. For example, the analyst might have found outputs and outcomes suggesting a relatively robust project, but also discovered that an alternative approach was expected soon to displace the project's technology, such that the outlook for long-term benefits was pessimistic. Or, there might be little in the data to suggest a robust project, but the case analyst might uncover a new development in the works that would give cause for optimism.

To facilitate translating the outlook descriptions into an indicator of commercial progress, the projects were divided into three groups with respect to outlook. Group 1 included those projects whose outlook the case-study analyst described as highly promising, or excellent, or on track. Group 2 included three subgroups: those whose outlook was described as neither particularly strong nor weak; those whose outlook was described as promising but with serious reservations or qualifications added; and those whose outlook was uncertain. Group 3 included those projects whose outlook was portrayed in clearly pessimistic terms. The CPRS was formulated to include a numerical outlook variable, reflecting the outlook group to which a project is assigned.

7.0 Applying Weighting Algorithms to the Selected Indicator Variables and Calculating Scores

Weights were assigned to each of the selected indicator variables to determine how they figure in the composite rating. As was the case with the other composite scoring systems reviewed in Section 7, expert judgment was used to determine the weights for the CPRS.

The range of values observed for each of the variables in the database compiled for the first 50 completed ATP projects influenced the specification of weights. An objective was to dampen the effect of outlier values. Applying the assigned weights to each indicator variable produced components of the raw score. Summing the components of the raw scores resulted in the composite raw score that was then converted to a star rating.

7.1 Knowledge Creation Scoring

Table 5 summarizes the weighting of indicator variables selected to indicate progress toward knowledge creation. Column 1 lists the variables in declining order of their assumed importance as indicators of knowledge creation. Column 2 shows the range of values for each variable observed for the first 50 completed ATP projects. Column 3 shows the weighting algorithm for each variable. Column 4 gives the range of raw scores calculated for the first 50 completed projects by applying the weighting algorithm to each indicator variable. The bottom row of the table shows the range of aggregated raw scores for knowledge creation.

7.1.1 Weighting the Technical-Award Indicator

Technical awards among the first 50 completed ATP projects ranged from zero to four in number. That is, some projects received as many as four awards from different organizations. Technical awards are assumed to serve as the best indicator that significant knowledge was created. The awarding of multiple awards by different organizations seemed largely independent of one another, and the decision was made to assign equal weights to each additional award. The raw score for technical awards is calculated by multiplying 1, the weight, times N, the number of science and technical awards received. The raw scores for awards ranged from 0 to 4 for the first 50 completed ATP projects.

7.1.2 Weighting the Patent Indicator

Patent filings ranged in number from 0 to 26 per project, with the range reflecting patenting strategy as well as the amount of knowledge created. That is, one project may file a single patent to capture its knowledge creation, while another may file many patents to capture a comparable or different amount of knowledge. The weighting algorithm values the first patent at half that of a technical award, and additional patents at a sharply decreasing rate. The raw score for patent filings is calculated as 0.5 times the square root of the number of patent filings. The raw score for patent filings ranged from 0 to 2.5 for the first 50 completed ATP projects.

7.1.3 Weighting the Publication/Presentation Indicator

Publications and presentations ranged in number from 0 to 214, with one project having far more than the rest. The weighting algorithm values the first publication or presentation the same as the first patent, and additional publications and presentations at an even more sharply decreasing rate. The raw score is calculated as 0.5 times the fourth root of the number of publications and presentations. The raw scores ranged from 0 to 1.9 for the first 50 completed ATP projects.

7.1.4 Weighting the Product/Process Indicator

Products and processes in the market or expected soon ranged in number from 0 to 5. Some projects had more than one product, some a combination of product and process. Where there were multiple products or processes for a project, it is assumed they all serve to indicate the same underlying body of knowledge creation. Hence, having multiple products and processes does not increase the raw score. The weighting algorithm is binary: 0 if there are no products or processes; 0.5 if there are. This assigns the same value to having any products and processes as to having a single patent or publication. The raw scores ranged from 0 to 0.5 for the first 50 completed ATP projects.

7.1.5 Aggregate Raw Scoring of Progress Toward Knowledge Creation

The aggregate score for each project is computed by summing across the scores for the four indicators. The aggregate raw scores for knowledge creation for the first 50 completed ATP projects ranged from 0 to 8.9.

7.1.6 Sensitivity Testing of Knowledge Creation Scores

Table 6 summarizes the results of sensitivity testing of the aggregated raw score for knowledge creation to changes in the values of each of the indicator variables. 30 An inspection of the tabular values reveals the marginal contribution of each indicator to the aggregate raw score for knowledge creation. For example, rows 1-4 show that the first technical award increases the raw score by 1.0; the first patent filing, 0.5; the first publication or presentation, 0.5; the first product or process, 0.5. If a project had one of each of the indicator variables, the raw score would be 2.5. Comparing rows 1-4 with rows 6-9 shows the marginal contribution of the second unit of each indicator. Receiving a second technical award adds 1.0 to the raw score; filing for a second patent adds 0.2 to the raw score; publishing or presenting a second paper adds 0.1 to the raw score; adding another product or process in the market adds 0.0 to the raw score. The weighting scheme moderates the effect of unusually high rates of publishing and patenting for one of the projects in the sample, as can be seen by comparing rows 17-19 with rows 2 and 3. Having 1 publication yields a score of 0.5, whereas having 200 yields a score of 1.9. Having 1 patent filing yields a score of 0.5, whereas having 20 patents yields a score of 2.2. Row 19 shows the effect of inserting values toward the upper end of the observed ranges for each of the indicator variables at once.

7.2 Knowledge Dissemination Scoring

Table 7 summarizes the application of weights to the variables selected to indicate knowledge dissemination. Column 1 lists the variables in declining order of their relative weights. Column 2 shows the range of values for each selected variable observed in the database. Column 3 shows the weighting algorithms. Column 4 gives the range of raw scores for each variable calculated by applying the weighting algorithm to the corresponding indicator variable. The bottom row of the table shows the range of aggregated raw scores for knowledge dissemination across the first 50 completed ATP projects.

7.2.1 Weighting the Publication/Presentation Indicator

A step function was used in weighting the dissemination value of publications and presentations. A single publication or presentation will result in a weight of 1. Additional units up to 10 add to the score at a sharply declining rate; and units in excess of 10 add at an even slower rate. The assumption is that when a project produces multiple publications and presentations they are closely related, and each subsequent publication or presentation does not convey as much information as the previous one. 31 The approach avoids having one project's extremely large number of publications/presentations overwhelm all other variables in the aggregate raw score. The starting range of 0 to 214 translates to a range of weighted raw scores of 0 to 4.6 after the application of the weighting algorithm.

7.2.2 Weighting the Patent Indicator

The step-function weighting algorithm for patent filings is identical to that for publications and presentations, based on a similar rationale that as the number of patents in a project increases, the contribution to knowledge dissemination of additional units declines. The range of weighted raw scores is 0 to 3.6.

7.2.3 Weighting the Collaboration Indicator

Three forms of collaborative relationships are tracked, namely (1) collaborations between award recipients and university researchers, (2) R&D collaborations among award recipients and other firms, state and federal laboratories, and other non-university organizations, and (3) collaborative ties between award recipients and other firms for technology commercialization. The weighting algorithm assigns a weight of 1 to each of these forms of collaborative relationship found in a project. The rationale is that each type of collaboration provides a different pathway of knowledge flows. Among projects in the sample of 50, some had no collaborative relationships; some had one, two, or three of the three forms listed. Hence, the range of weighted raw scores is 0 to 3.

7.2.4 Weighting the Product/Process Indicator

The weighting algorithm for products and processes effectively treats products and processes as half as important as patents and publications/presentations as knowledge disseminators. The rationale for the lower weight is the additional effort and difficulty in extracting knowledge by inspection and reverse engineering of product and processes. The weighted raw score ranges from 0 to 1.1, depending on the number of products and processes.

7.2.5 Weighting the Technical Award Indicator

The weighting algorithm for receipt of technical awards effectively treats technical awards as one-fourth as important as patents and publications/presentations. The weighting algorithm reflects the fact that technical awards raise awareness of a new technology, and hence, further disseminate knowledge, but the award does not itself typically convey much detailed knowledge. Since multiple awards may call greater attention to the new technology, the weight increases slightly with increasing numbers of awards. The weighted raw score ranges from 0 to 0.5, based on awards that range from 0 to 4.

7.2.6 Aggregate Raw Scoring of Progress Toward Knowledge Dissemination

The aggregated weighted raw score for knowledge dissemination ranges from 0.7 to 12.8. The lower end of the range is positive, despite the fact that the lower end of the ranges for each of the component indicator variables is 0, and, further, that the low-end of the range for the aggregate weighted raw score for knowledge creation is 0. This may seem a contradiction, but is not for two reasons. (1) The knowledge dissemination score contains collaboration as an indicator of progress, and one of the joint-venture projects in the sample exhibited none of the measured indicators except collaborative activity. (2) Although one or more projects had 0 values for each of the other variables, no project had 0 values for all of the variables.

7.2.7 Sensitivity Testing of Knowledge Dissemination Scores

Table 8 summarizes sensitivity testing of the aggregated raw score for knowledge dissemination to changes in the values of indicator variables. Rows 1 through 5 show the value of the first unit of each variable, with a single publication or presentation resulting in a score of 1.0; a single patent, 1.0; a single type of collaboration, 1.0; a single product or process, 0.5; and a single technical award, 0.3. Row 6 shows that if one of each of the indicators were obtained, a raw score of 3.8 results.

Comparing rows 7-12 with 1-6 shows the marginal effect of adding the second unit of each variable. The second publication or presentation adds 0.4 to the score. The second patent filed adds 0.4 to the score. The second type of collaboration adds 1.0 to the score. The second product or process adds 0.2, and the second technical award adds 0.1. Having two versus one of each of the indicators results in a score of 5.9, compared with 3.8.

The small differences revealed by comparing rows 13, 14, and 15 reflect the step function used in the weighting algorithm for publications (and also patents). Increasing the number of publications and presentations from 10 to 11 increases the raw score by only 0.1. Increasing the number from 11 to 200 increases the raw score by only 1.2. Rows 15-19 show the resulting raw score if values near the upper end of the range are used in turn for each of the indicator variables. Row 20 shows the resulting raw score of 12.6 from using values near the upper end of the range for all of the indicator variables at once.

7.3 Commercialization Progress Scoring

Table 9 summarizes the application of weights to the variables selected to indicate commercialization progress. Column 1 lists the four variables used to indicate progress toward commercialization. Each of the variables used has a potentially substantial impact in the scoring. Column 2 shows the range of values observed for each indicator variable in the database of the first 50 completed ATP projects. Column 3 presents the weighting algorithms applied to the indicator variables to calculate the raw scores, shown in column 4. The bottom row shows the range of aggregated raw scores for commercialization.

7.3.1 Weighting the Product/Process Indicator

Products and processes in the market now or expected soon are given a relatively heavy weight, because it indicates the likelihood that a project has progressed to a stage in which economic benefits may result. A primary interest is that there be a product or process. The weighting algorithm reflects the relatively large importance attached to this indicator, with the first product/process receiving a score of 4.25. Additional products/processes contribute less. The raw score ranges from 0 to 5.8.

7.3.2 Weighting the Capital Attraction Indicator

Capital attraction indicates that additional resources are being made available for further development and commercial efforts. Due to data difficulties discussed earlier, a single-value weight is assigned to a project if it has attracted funding from one or more of the sources identified, regardless of amount. The indictor variable is assigned a weight of 0 if a project has attracted no additional funding, and a weight of 3 if it has attracted funding from any of the identified sources.

7.3.3 Weighting the Employment Gains Indicator

As indicated earlier, employment data were compiled only for the small, single-applicant companies. Because most of these companies were very small at project start, the percentage increases for a number of the projects tended to be very large. Growth rates of 50% were routine, and rates approached 2000% for several projects. For this reason, the weighting strategy for small companies is to treat rates of employment change up to 50% as the norm, assigning a weight of 0 for rates of change of 50% or less. If employment increased by more than 50%, a weight of 2.5 times the fourth root of the gain in excess of 50% is assigned, providing a relatively large weight, while preventing the extremely large gains in several cases from over­whelming other indicators. In the case of bankruptcy-either of a small company, single applicant, or a leader of a joint venture-a negative weight of -6 is assigned to signal that there is a serious impediment to commercial progress through the direct path of the innovating companies. In cases other than bankruptcy, a default weight of 1.5 is assigned to projects for which employment data were not collected (i.e., large companies and joint ventures). Applying the weighting algorithm resulted in raw scores ranging from -6 to 5.2.

7.3.4 Weighting the Business Awards Indicator

The weighting algorithm for business awards is to assign 0 if there are no awards and 3.25 for one award. Additional awards add 0.25 each to the score. The weighted raw scores range from 0 to 3.8.

7.3.5 Weighting the Outlook Indicator

As indicated earlier, the subjective information in the cases was used to group the projects into three groups by outlook. The weighting strategy assigns a raw score of +4 to those in group 1, whose outlook is strong; 0 to those in group 2, whose outlook is neither clearly strong nor poor; and -4 to those in group 3, whose outlook is poor.

7.3.6 Aggregate Raw Scoring of Progress Toward Commercialization

The aggregated weighted raw scores for commercialization range from -10 to 21.7. The range for this score is much wider than for knowledge creation and dissemination scoring, and may be negative if the company goes out of business, or if the outlook is poor, or if both occur.

Scoring at the low end of the range are projects conducted by companies that went bankrupt and for which the outlook revealed no active alternative champion of the technology to take it forward. At the high end were projects conducted by award-winning businesses-particularly those that are fast growing-with commercialized products or processes based on the technology, available resources to continue development and commercialization of the technology, and an outlook for continued robust progress.

Table 10 summarizes the results of testing the sensitivity of the aggregate raw score for commercialization to changes in the value of each of the indicators, and to several changes in com­bination. The use of certain combinations of values here and in the previous sensitivity testing is not to imply that they are expected to occur in the combination shown, but rather the intention is to test the sensitivity of results to extreme values.

For the purpose of clearer exposition, two of the variables, employment gains and out­look, are entered in descriptive terms rather than in terms of the numerical value into which the description is translated. The model includes four possible conditions of employment gain, only one of which allows the score to change as a function of the amount of increase in the gain. It contains three possible outlook states.

The sensitivity testing begins with the condition of employment gain. Rows 1-3 show the effect of holding the outlook constant at Group 2 (neither clearly positive or negative and having a 0 effect on the raw score), holding other variables except employment gains constant at 0, and changing only the condition of employment gains. A joint venture or large company for which meaningful employment data were not collected receives a default score of 1.5. A small company, drawn from a group in which up to 50% employment gains were commonplace, receives a score of 0. Beyond a 50% employment gain, the score is positive and increasing at a decreasing rate, with a 100% gain receiving a score of 2.1, and a 200% gain, a score of 2.8.

Comparing rows 1 and 5 reveals the effect of changing only the outlook from Group 2 (neither clearly positive nor negative) to Group 1 (positive). The raw score increases by 4.0, showing a relatively large impact on scoring of the outlook.

Comparing rows 6-8 with row 1 shows the contribution to the raw score of the first unit of each of the other indicator variables. Commercializing a product or process adds 4.3 to the raw score; attracting additional capital adds 3.0; receiving a business award adds 3.3. Row 9 shows a raw score of 16 resulting from the coincidence of a joint venture or large company having one unit of each indicator variable, combined with a positive outlook.

Comparing row 10 with row 6 reveals the effect of commercializing a second product or process to be 0.5. Comparing row 11 with row 8 reveals the effect of a second business award to be 0.2. Because capital attraction is modeled as either yes or no, it is not tested for other possible values.

Comparing rows 12 and 13 with row 3 isolates the effect of changing only the outlook. There is an 8 point drop in the raw score as the outlook changes from positive to negative.

Note: JV denotes a joint-venture project; Co. abbreviates company; and Bnkrpt abbreviates bankruptcy. Employment-gains data were unavailable for joint ventures and large companies, and a proxy value of 1.5 is used in lieu of real data. Group 1 had a positive outlook, Group 2 a neutral, clouded, or indeterminate outlook, and Group 3 a poor outlook.

Rows 12, 14, and 15 combine increasing changes in small-company employment with a positive outlook, producing raw scores of 6.1 for a 100% employment gain/positive outlook,7.6 for a 500% gain/positive outlook, and 9.3, for a 2000% gain/positive outlook. In contrast, row 16 combines a 100% loss in employment, signaling bankruptcy, with a negative outlook, producing a raw score of -10.

The last row of the table, row 17, combines strong performance values for each of the five commercialization indicator variables, producing a raw score of 21.8. One of the projects achieved a score very close to this.

7.4 Composite Scoring and Star Rating

The composite raw score is calculated for each project in the portfolio by summing the project's raw scores for knowledge creation, knowledge dissemination, and commercial progress. The composite raw score is factored by 6 to facilitate dividing the projects into five groups of adjusted composite scores. The group with a score of 4 or higher receives the highest rating of 4 stars; a score less than 4 but at least 3, 3 stars; less than 3 but at least 2, 2 stars; less than 2 but at least 1, 1 star; and less than 1, 0 stars.

7.4.1 Range of Ratings for First 50 Completed Projects

For the first 50 completed ATP projects, the computed composite raw scores ranged from -9.0 to 30.8, and the range of adjusted composite scores ranged from 0 to 5. The number of stars assigned these 50 projects ranged from 0 to 4.

7.4.2 Sensitivity Testing of Composite Scores and Star Ratings

This section adds to the previous sensitivity testing by examining how the overall scores and star ratings change in response to alternative input values of indicator variables. As a starting point, consider the raw scores necessary to produce the various star ratings. A 4-star rating requires a composite raw score of 24 or greater. A 3-star rating requires a composite raw score of at least 18 but less than 24. A 2-star rating requires a composite raw score of at least 12 but less than 18. A 1-star rating requires a composite raw score of at least 6, but less than 12. And, a 0-star rating requires a composite raw score less than 6.

Table 11 shows the composite scores and star ratings associated with different combinations of values for the indicator variables. To simplify comparisons, all seven hypothetical cases presented are based on a very small single-applicant firm. A degree of collaboration is assumed for all cases, because most companies in the ATP, particularly very small companies, have some form of collaboration.

The first case shows early publication and patenting progress, but no commercial follow-through and a cloudy outlook casting doubt on further progress. It receives a zero star rating. The second case has the same outputs as the first case, but it has a favorable outlook that boosts its star rating to 1. The third case shows some progress across the board and a favorable outlook, but only modest firm growth (from 2 to 4 persons). It receives a moderate star rating of 2. The fourth case shows no patents or publications but relatively active progress on the commercial side, more robust company growth (from 2 to 12 persons), more collaboration, and a favorable outlook. It receives a relatively robust score of 3. The fifth case is essentially the same as the fourth case, except that it substitutes knowledge outputs for one of the commercial outputs. Like the preceding case, it receives a star rating of 3. The sixth case is the same as the fifth case, except that it adds technical and business awards. It receives the highest star rating of 4. The seventh case strengthens the outputs and collaboration, but drops the awards and changes the outlook from favorable to unfavorable. The star rating drops to 2 despite the past accomplishment. These cases show that different combinations of output, out­ come, and outlook data can produce the same or different composite ratings.

7.4.3 CPRS Calculation Database

The CPRS is implemented via an Access database with the weighting algorithms embedded in calculation queries. Values of the indicator variables for each project are entered into a form, along with other non-CPRS project data. Data for additional completed projects are added as the portfolio grows. All indicator variables, CPRS adjusted scores, and corresponding star ratings can then be analyzed by building queries and running reports.

7.4.4 Illustration of Four-Star and Zero-Star Projects

Table 12 summarizes scoring information for a project receiving 4 stars. The project was an information technology project conducted by a small company, Engineering Animation, Inc. The company won technical and business awards, and had many collaborations, large employment growth, and several products on the market. Table 13 summarizes scoring information for a project receiving no stars. The project was an electronics joint venture project led by a small company, Hampshire Instruments, Inc., which went bankrupt just after the project completed.

7.4.5 Distribution of First 50 Completed ATP Projects by Performance

Figure 2 shows the distribution of the first 50 completed ATP projects by performance as scored by the CPRS. As may be seen, the largest group of projects, 32 percent, scored in the two-star category-accomplishments, but not particularly robust progress overall. Twenty-six percent scored in the bottom category (1 star or less). Sixteen percent scored in the top category, receiving 4-stars, while an additional 26 percent also showed relatively robust progress, scoring in the 3-star category. These results are consistent with the program's expectation that not all the project will be strong performers, given the challenging nature of their undertakings.

Figure 2. Distribution of Completed ATP Projects by CPRS

TABLE 12 - Four-Star Project (Engineering Animation, Inc.)

Those projects with one or no stars were generally those that showed few outward signs of progress (in terms of the indicator metrics) toward contributing to technology creation, dissemination, and commercialization. This group also included those that showed early signs of progress, but then faltered. The two-star category generally included projects that showed modest but no overly robust signs of progress and projects that had shown progress but whose future prospects seemed clouded or unfavorable. The two-star category also may include technologies that are just slow to develop or that take more time to develop than allowed by the assessment time frame. The three- and four-star categories included projects that made sustained progress, continuing into commercialization, with favorable prospects for the future.

8.0 Critique of CPRS

The CPRS is a management and communications tool particularly useful for providing an easy-to-grasp assessment of project and portfolio performance in the intermediate period of a public R&D program. The tool facilitates bridging from the project level to the portfolio level. The CPRS can be practically implemented as demonstrated by ATP's experience. It is intended to distinguish among varying degrees of progress toward achieving a program's multiple goals based on indicator metrics that relate to those goals.

8.1 Data Limitations

The CPRS was specifically formulated to use indicator variables available from ATP's case studies of its first 50 completed projects. Once developed, the CPRS can be exercised with new data derived either from case studies or from surveys such as ATP's Business Reporting Survey (BRS), provided data compatibility is maintained. ³² The formulation of the CPRS to use available data meant that it would be practical to implement, but it also meant that the system might use less than ideal data. For example, the available data on publications were simple counts, whereas, data adjusted for quality and significance might provide a better indicator of knowledge creation and dissemination. Similarly, available data on capital attraction indicated the source but not consistently the amount of funding; yet, resource amount is likely more important than funding source as an indicator of commercial progress. The available data on products and processes at the time the CPRS was formulated identified products by name and description, but did not consistently provide sales volumes or commercialized value; yet, if market data had been consistently available, it would have provided a stronger indicator of progress toward commercialization than a count of products and processes in the market or expected soon.

Some relevant categories of data were missing altogether. For example, the available data did not provide an indicator of knowledge creation embodied as human capital only, that is, knowledge that may reside in the minds of research staff and may show up at a later time in outputs associated with other efforts. For some programs that emphasize university research, a measure sometimes used to capture human capital is the number of graduate students trained, but this measure is not generally applicable to ATP projects and was not captured in the case-study data. To the extent that knowledge is created and embodied only in the minds of researchers, it is omitted from the CPRS formulation even though it is potentially important.

"Outlook" is highly subjective, and the analysts may not have consistently captured outlook over the same time periods for all the projects. In some cases, case analysts spoke of the future in terms of relatively short-term events; in other cases, they spoke of future prospects in more sweeping terms. Possible variability in time horizons covered by outlook calls into question the actual time horizon covered by the CPRS.

In short, there may be other variables, or refined specifications of variables, that would better indicate progress toward program goals during the intermediate period than those specified for the current CPRS formulation. On the other hand, those variables used have proven feasible to compile.

8.2 Methodological Issues

There are methodological issues associated with construction of composite ratings. The construction of the CPRS is ad hoc and improvisational, reflective of the absence of underlying theory to guide composite ratings. On the other hand, there is precedence for developing empirically based composite rating systems and for using expert judgment to assign weights to the selected indicator variables. The selection of indicator variables and the weighting algorithms specified in the CPRS are based on expert judgment informed and constrained by observations of actual data from the first 50 ATP completed projects. There is a lack of empirical verification of the relationships modeled; problems may lie with the formulation. For example, perhaps one goal should have received more or less weight relative to another than is built into the model. Perhaps one indicator should be given more or less weight than another in the scoring of progress toward a given goal. At this time, there is no way to know.

Construction of the CPRS entails aggregation of diverse data. Although we have learned that apples and oranges cannot be added, general pieces of fruit can be. That is, shifting the category up a level can reduce the incompatibility problem. Here we define the variables as indicators of progress toward a common set of goals. These indicators are related; combining them does not result in a single measure that is incomprehensible.

8.3 An Inexact Measure

Due to the limitations discussed above, CPRS ratings should be viewed as roughly indicative of overall project performance in terms of progress toward intermediate program goals. It should be noted that the rating system sorts projects into high, medium, and low performers; it does not provide dollar estimates of their contributions to long-run national economic benefits. For this reason, projects with the same composite ratings are not necessarily equal in their potential to deliver long-run benefits. Projects with similar ratings have comparable composite levels of outputs/outcomes/outlooks during the relevant time, but their actual economic impacts may differ substantially.

A high rating signals strong expectations about a project's progress toward contributing to ATP's goals. A low rating casts doubt on that expectation. But in neither case do the ratings rule out the possibility of surprise. The ratings are based on information compiled at a point in time. Projects advance at differing rates. Future developments could alter expectations about a project's long-run success.

A further point to note is that the rating system does not incorporate a separate measure of the role of ATP in the score, that is, it does not separate out that part of progress that is directly attributable to ATP. ATP, for example, may cause a technology to be developed that otherwise would not have been developed. It may accelerate technology development by a given number of years. It may change the scale and scope of projects. The CPRS examines project progress against program goals per se. Data on ATP's role has been collected by status-report case studies and by survey for most completed projects thus far, but attempting to combine a measure of ATP's role with project progress indicator data did not seem feasible. Data on the role of ATP, presented as a separate factor, can be viewed in conjunction with the performance ratings. ³³

To the extent that a project's performance radically changes in the out years in ways not captured by the case-study analyst's outlook, the CPRS ratings will be a poor predictor of a project's longer-run performance. Furthermore, to the extent that a project with little in the way of the outputs/outcomes measured has at least one output/outcome that eventually yields unusually high benefits, the ratings will not have good predictive value. Consider, for example, the case where a project's only measured output/outcome is a single publication that some years later has a profound impact on another organization's accomplishment. The project's performance rating will have been low, yet its ultimate benefits may be high. This possibility suggests that one should be careful not to dismiss low-scoring projects prematurely. On the other hand, it is thought that most projects that have produced few if any of the specified indicators throughout the project and extending 3 to 4 years into the post-project period will be unlikely to suddenly bloom.

A potential test of the predictive value of the rating system would be to monitor a selection of projects in each performance category over time. One such test would be to determine whether the four-star projects continue to progress and deliver significant benefits at a higher rate than the 0- and 1-star projects. ³⁴

The long-term predictive value of the CPRS for 2-star performers may be more difficult to determine, mainly because the category includes projects with three different types of outlook, all of which receive a neutral rating: (1) those whose future outlook was considered neither strongly positive nor strongly negative, (2) those for which there were both positive and negative elements which seemed largely offsetting, and (3) those for which the outlook was considered too uncertain to call. Hence, it would not be counter to the rating system's findings if a 2-star performer eventually emerged as either a highly successful project, a largely unsuccessful project, or remained a moderate performer. It may be possible to refine the CPRS to better distinguish across projects with the different types of outlook. One step would provide a finer breakout of outlook categories in the CPRS scoring; a related step would support the finer breakout by improving the quality of outlook data provided by case-study writers.

9.0 Summary and Conclusions

This report has presented a new framework of evaluation, together with a new evaluation tool embedded in the framework. Together, the framework and the tool boost the potency of the case study method—one of the mainstays of program evaluation. The result is an evaluation methodology that allows program administrators and project managers to bridge from individual project case study to portfolio analysis, and to answer a question of central importance to public policy makers and other stakeholders, namely, how are projects in ATP's portfolio performing overall in the intermediate period against ATP's mission-driven multiple goals? The methodology provides a practical tool that can facilitate a deeper understanding of a program's portfolio of funded projects, and yet convey an easy-to-grasp measure of overall performance.

Case study is just one of a set of evaluation methods that ATP and most other public R&D programs use for assessment. The framework and new tool presented here are rooted in case study and, hence, stand on the shoulders of an existing approach.

ATP developed the framework partly as a result of foresight and partly through a series of evolutionary steps. The effect was to move from conducting single-project and cluster-of-project case studies presented individually, to defining a workable portfolio of projects, all of which would receive a "mini case study" and be subject to uniform collection of a set of progress indicator data. The period of focus was after project completion and prior to long- term benefits realization. Detailed economic case studies for a subset of the projects in the portfolio allow the estimation of minimum net benefits for the portfolio or the program. Aggregation of the indicator data by category shows outputs related to each program goal. Development of a composite performance rating system (CPRS) allows the indicator data to be combined to provide an easy-to-grasp overall performance measure across multiple pro­gram goals. The distribution of CPRS ratings within the portfolio gives program administrators a handle on the overall performance of the portfolio and an easy way to communicate that performance. At the same time, linking the composite ratings back to the individual case studies facilitates further investigation into the impact of funded projects.

The CPRS tool is still in a prototype development stage for application specifically to ATP. It is undergoing review and critique for possible improvements or extensions. At the same time, the CPRS has been used extensively to monitor project and portfolio performance during the intermediate period after project completion and before long-term benefits have had time to be realized and measured. It was used to rate ATP's first 50 completed projects and is being used to rate the next group of completed projects. ATP has used the CPRS tool to brief ATP oversight and advisory bodies, public policy analysts, evaluation groups, the broader S&T community, and the general public about ATP's performance.

There is considerable precedence for using composite scoring as a management tool. Composite rating systems have recently been developed or proposed for use by other federal agencies, international bodies, hospitals, and businesses, several examples of which are presented in the report. For example, a composite rating was developed to score the stability of financial institutions of variable size, location, and other characteristics to meet federal guidelines. In the prior cases examined, a composite rating system was developed to make complex information about multiple aspects of an issue more understandable to program administrators and other stakeholders in order to facilitate decision making.

A common characteristic of composite rating systems is the lack of an existing theoretical basis on which to base its development. The counterpart composite ratings examined were unavoidably ad hoc in nature, based on empirical experience rather than an existing theory or literature. Like the CPRS, the other rating systems examined relied heavily on expert judgment to select the indicator variables used for the composite measure and to assign weights to the indicator variables to determine how much each would count in the composite measure. This is a methodological limitation that may be reduced over time by further analysis and definition of underlying, functional relationships between alternative indicator metrics and the goals to which they relate.

The CPRS was developed after the first 50 mini case studies of completed ATP projects had been completed, and it was formulated specifically to use the data on outputs, outcomes, and outlook uniformly collected in those case studies. The available data were examined and those variables that appeared best to serve as indicators for each of three program goals were selected. Weights were assigned to the indicator variables according to expert judgment of the analyst in consultation with program administrators. The CPRS was designed to measure overall pro g re s s during the intermediate performance period toward accomplishing the following three goals: creating knowledge, disseminating the knowledge, and commercializing the technologies created from the knowledge base. It was not intended to provide a measure of long-run economic benefits. Given that the CPRS does not provide a measure of net benefits, and, in any case, project performance may change after the case-study data are collected, projects with similar CPRS ratings may differ in their long-term net economic benefits.

By weighting and combining progress indicators, a star rating was computed that provides a composite view of each project's progress overall during the interim period toward accomplishing program mission. From the individual project ratings a distribution of star ratings for the portfolio was computed to provide an overview of performance across the whole portfolio. By reducing a large amount of detail to a single symbolic rating for each project-0 to 4 stars-the CPRS conveys a snapshot of project performance. By reducing an even larger amount of often conflicting detail to a distribution of symbolic ratings across the portfolio-16% with 4 stars, 26% with 3 stars, 34% with 2 stars, and 24% with 1 or no stars-the CPRS conveys an immediate picture of portfolio performance. At the same time, all the details are preserved in the underlying project case studies to allow one to probe the specifics of each project.

The CPRS is consistent with the idea that there are varying degrees of project success that can be distinguished at a given time and signaled by indicator metrics. It is also consistent with the idea that cumulative project accomplishments at each stage-beginning with knowledge creation, continuing with knowledge dissemination, and progressing further with commercialization represent an increasing degree of project success. Though surely an imperfect measure, the CPRS as formulated distinguishes among projects in ATP's portfolio in terms directly tied to the program's mission-driven goals and that are meaningful to its stakeholders. It provides a composite performance measure that is practical to construct and easy to understand and communicate.

The CPRS presented here has been custom designed for ATP's application and would not be suitable for direct transference to other public R&D programs with different goals and different time horizons. However, the CPRS concept and the eight-step framework of which it is an element can be adapted to fit other programs.

The CPRS was developed in 2000-2001 using the first 50 ATP projects completed in conjunction with the writing of status reports (mini-case studies) on these projects. Since then, ATP's Economic Assessment Office has computed CPRS ratings and published over 100 additional status reports. All completed status reports and CPRS ratings can be accessed on a searchable web site (http://statusreports.atp.nist.gov/) and in the following publications:

•   Performance of 50 Completed ATP Projects: Status Report - Number 2, NIST SP 950-2, 2001
•   Performance of 50 Completed ATP Projects: Status Report - Number 3, NIST SP 950-3, 2006

19.  See Smith (2001).

20.  Kilbridge et al. (2001). The report may be viewed at www.fcg.com by clicking on the report cover.

21.  More about the Dow Theory Forecast and the Quadrix Stock-Rating System can be found at the newsletter Web site, www.dowtheory.com.

22. Geisler (2000), pp. 257-262.

23.  Geisler (2000), p. 257.

24. Geisler (2000), p. 258.

25. The table suggests well-defined stages of project/technology development, but in practice there are overlaps and non-linearities. A patent may be filed early in a research project, a spin-off product opportunity may occur early; opportunities for research publications may occur at various times during research and technology development, sometimes overlapping commercialization activities. The table is intended only to suggest tendencies for certain activities to concentrate during certain periods.

26. The items to be collected through the mimi case studies of completed projects were worked out between ATP's evaluation staff and the contractor engaged to perform the first status reports, taking into account program goals and feasibility of collecting the data.

27. Reports in fulfillment of the Government Performance and Results Act (GPRA) included a variety of metrics such as patents and projects under commercialization. In addition, results of economic case studies and econometric studies contributed to GPRA reporting.

28.  Geisler (2000), p. 255.

29. See Chapter 2, Section E ("Project Narrative") of the ATP Proposal Preparation Kit, obtainable by request from ATP by e-mail at atp@nist.gov or by phone at 1-800-ATP-FUND, and also available on ATP's Web site at www.atp.nist.gov.

30. Use of particular combinations of values for the indicators does not imply that those combinations are likely to occur in real projects. At the same time, all the values used in the testing are within the ranges observed in the first 50 completed ATP projects.

31. At the same time, it could be argued to the contrary that multiple publications and presentations may exhibit a critical mass effect, where the dissemination value increases at an increasing rate rather than at a decreasing rate as the number increases. This is an example of an area that could benefit from further research.

32. For a description of the BRS and findings from that survey, see Powell and Lellock (2000). The BRS is a primarily electronically administered survey of ATP project participants that has a post-project component that collects survey data overlapping to some extent the data collected by the status reports. It may be used alternatively to provide part of the data needed to implement the CPRS. Without the accompanying case-study approach, however, the framework described in Part I and the linkages between the distribution of CPRS ratings for the portfolio and the published detailed case studies of projects comprising the portfolio will not be realized.

33. See, for example, the table on ATP effects reported in ATP (2001), p. 24.

34. In fact, there has already been a partial test of the predictive value of the rating system. Data for 38 of the 50 completed ATP projects in the sample used to develop the CPRS had been collected by Long in 1997 (see Long 1999). Ruegg made a check four years later, in conjunction with preparing the overview chapter for the new status report (see ATP 2001) of top-rated projects from the earlier group. The check showed them all to be continuing their relatively strong performance in terms of further commercial progress.

35 Status Report - Number 2 contains all projects from Status Report - Number 1 (38 projects), as well as the additional 12 projects used in the CPRS formulation.

Return to Table of Contents or go to next section.

Date created: June 1, 2006
Last updated: January 3, 2007