NIST Advanced Technology Program
Return to ATP Home Page
ATP Historical Award Statistics Business Reporting System Surveys EAO Economic Studies and Survey Results ATP Factsheets ATP Completed Projects Status Reports EAO Home Page
Cover Page to GCR03-857

A Toolkit for Evaluating Public R&D Investment
Models, Methods, and Findings from ATP's First Decade

Rosalie Ruegg
Irwin Feller

July 2003

Download in Adobe PDF front section, Part I, Part II and Part III.

Note from Project Manager  [View PDF]

About the Authors [View PDF]

Acknowledgments [View PDF]

Abstract [View PDF]

Executive Summary [View PDF]

  • Evaluation Underpinnings
  • Multi-Faceted Methodological Approach
  • An Emerging Body of Findings
  • Firm/Industry Effects
  • Collaboration Effects
  • Spillover Effects
  • Interfaces with State Programs and Comparison with Counterpart Programs Abroad
  • Overall ATP Performance
  • Recommendations for Future Directions

Acronyms and Abbreviations [View PDF]

Introduction [View PDF]

  • The Political Economy of the Advanced Technology Program
  • The Role of Evaluation


Chapter 1. Evaluation Fundamentals

  • Objectives of Evaluation
  • Mapping Evaluation to Mission and Stages of Implementation: A Generic Evaluation Logic Model
  • Evaluation Steps
  • Best Practices

Chapter 2. Choosing Methods of Evaluation

  • Analytical/Conceptual Methods for Modeling and Informing Underlying Program Theory
  • Survey Method
  • Case Study: Descriptive
  • Case Study: Economic Estimation
  • Sociometric/Social Network Analysis
  • Bibliometrics: Counting, Citing, and Analyzing Content of Documents
  • Historical Tracing
  • Expert Judgment
  • Suggested Readings on Evaluation Methods

Chapter 3. ATP’s Evaluation Program

  • Background: Evaluation Drivers
  • ATP’s Evaluation Logic Model
  • Conceptual Tests of ATP’s Success
  • ATP’s Approach to Evaluation


Chapter 4. Modeling and Informing Underlying Program Theory

  • Concepts, Models, Metrics, and Paths Connecting Program Activities to Intended Impacts
  • Conditioning Expectations through Studies of Private-Sector Behavior
  • Conditioning Expectations through Studies of Other Public Sector Programs
  • Testing Dominant Paradigms
  • Summary of Research Informing Underlying Program Theory

Chapter 5. Survey Method

  • Gaining Early and Broad Perspective of a Program’s Effects
  • Extending and Deepening Survey Data on Program Effects
  • Establishing Routine Project Reporting by Electronic Survey
  • Soliciting Feedback by Survey on Customer Satisfaction and Marketing Issues
  • Using Survey for Case Study and to Address Research Questions
  • Summary of ATP’s Use of the Survey Method

Chapter 6. Case Study Method

  • Economic Case Study of Individual Projects
  • Project and Portfolio Assessment Using Multiple Cases Studies with Uniform Collection of Key Indicator Data
  • Explicating Program Features and Exploring Program Dynamics
  • Summary of ATP’s Use of the Case Study Method

Chapter 7. Econometric/Statistical Method

  • Testing ATP’s Leveraging Effects on Advanced Technology Development
  • Modeling Impacts of Public-Private Partnerships on Firm Productivity
  • Spillovers, Appropriability, and Firm Productivity
  • Analyzing the Role of Universities in Public-Private Partnerships
  • Modeling the Impact of Publishing by Industry Scientists on the Quality of Innovative Output
  • Investigating Characteristics and Impacts of Joint Ventures
  • Summary of Econometric/Statistical Methods

Chapter 8. Other Evaluation Methods Used by ATP

  • Expert Judgment
  • Bibliometrics
  • Emerging Methods: Using Social Network Analysis to Identify Knowledge Spillovers
  • Emerging Method: Using the Cost Index Method to Estimate Social Benefits
  • Emerging Method: Using a Composite Performance Rating System Constructed from Case Study-Derived “Indicator” Data
  • Summary of Other Evaluation Methods


Chapter 9. A Crosscutting Look at Study Findings

  • Direct Private Firm Effects
  • Collaboration Effects
  • Spillover Effects
  • State and International Technology Programs
  • Measures of Overall ATP Performance
  • Summary of Crosscutting Findings

Chapter 10. Conclusions and Recommendations

  • Program Evaluation: An Essential Tool for ATP
  • Opportunities and Future Directions
  • Closing Note


About the Advanced Technology Program (inside back cover)



Date created: February 9, 2004
Last updated:
August 17, 2009

Return to ATP Home Page

ATP website comments:  / Technical ATP inquiries:

NIST is an agency of the U.S. Commerce Department
Privacy policy / Security Notice / Accessibility Statement / Disclaimer / Freedom of Information Act (FOIA) /
No Fear Act Policy / NIST Information Quallity Standards / (performance of federal programs)

Return to NIST Home Page
Return to ATP Home Page Return to NIST Home Page Go to the NIST Home Page