What
is Program Evaluation and Why is it Important?
Program
Evaluation is the systematic assessment
of program implementation and results that are valid,
reliable, and credible. It consists of various activities:
- needs assessment,
- design assessment,
- process/implementation,
- choice of methodologies,
- outcome and
impact.
The Advanced
Technology Program (ATP), since its birth
in 1990, has had an active evaluation program, with
a dedicated staff and budget. With the implementation
of the Government Performance and Results Act (GPRA)
in 1994, ATP was (and is) a leader in using several
methods of evaluating the program and using the findings
in computing performance metrics. In 2002, the President’s
Management Agenda added additional requirements
aimed at improving performance management practices
of federal agencies. These requirements, known as
the Program Assessment Rating Tool, or PART, include
new investment criteria for federal R&D programs.
The Office of Management and Budget (OMB) and individual
government programs perform PART scoring separately.
What is
the Program Assessment Rating Tool (PART)?
The Program
Assessment Rating Tool (PART) identifies a program’s
strengths and weaknesses and informs funding and management
decisions aimed at making the program more effective.
OMB evaluated
ATP in 2002. ATP met with OMB staff to present evaluation
results and ATP’s proposed scores for each component.
OMB agreed that ATP excels in program management and
strategic planning
Program
Purpose and Design: (ATP Score:
80%; OMB Score: 20%)
ATP assigned
a score of 80% to this component citing that the proposed
reforms submitted to Congress in 2001 would improve the
program. OMB scored ATP at 20% citing “there is
an adequate amount of funding from private sources.”
Strategic
Planning: (ATP Score: 100%; OMB
Score: 86%)
OMB rated this component
very high except for whether ATP collaborates and coordinates
effectively with related programs that share similar
goals and objectives.
Program
Management: (ATP Score: 100%; OMB
Score: 100%)
OMB gave ATP a perfect
score for this component, highlighting the excellent
management of the program.
Program
Results: (ATP Score: 100%; OMB
Score: 67%)
OMB rated this component
in the mid-range citing that impact studies are primarily
prospective and that outcomes and impacts are difficult
to measure.
Sources:
- The Advanced
Technology Program: Reform with
a Purpose, February 2002, The
Advanced Technology Program: Reform with
a Purpose, www.atp.nist.gov/atp/secy_rept/contents.htm
- A
Toolkit for Evaluating Public R&D Investment: Models, Methods,
and Findings from ATP’s First Decade: Models,
Methods, and Findings from ATP’s First Decade,
Irwin Feller and Rosalie Ruegg, (NIST GCR 03-857).
- Federal Evaluators,
materials prepared for presentation to OMB Examiners,
Spring 2005
- Office of Management
and Budget, Program Assessment Rating Tool (PART), http://www.whitehouse.gov/omb/part/print/index.html
Factsheet 1.A3 (Prepared
by Jeanne Powell and Stephanie Shipp, April 18, 2005) |
