What characterizes impact and outcome evaluations? How do they differ? How are they used in professional practice?  At the outset, some of the answers seemed straightforward: Impact evaluation assesses short term program outcomes, and outcome evaluation assesses the long term reach of the program (PACE mySPH, 2012, December 11).  Yet, the more each evaluation approach was examined, the more difficult and confusing the answers became, as each resource defined the approaches, and how they were used in practice differently, and in some instances used similar definitions in each account of impact and outcome evaluation in practice.

Impact Evaluation

The Organisation for Economic Co-operation and Development (OECD, n.d.) suggests that impact evaluation be used at the outset of program planning, for innovative or new programs, to assess whether or not the programs are meeting the short-term needs of the participants, or to determine which aspects of a program are working or not.  Contrastingly, the Centers for Disease Control and Prevention (CDCa, n.d; CDCb, n.d.) suggest that impact evaluations are utilized to determine the widespread outcomes of a program (community transformation), by using statistics collected over time to determine how effective the intervention has been (i.e.: to determine if a pre-natal care community public health initiative is positively affecting change on a community, infant mortality rates, rates and severity of infant/childhood illness, etc. are assessed).  Interestingly, the CDC contradicts itself when it suggests that impact evaluation is used at the end of a program, to measure the long-term impacts of a program on a community (CDCb, n.d.), and that it is used at intervals when a program is ongoing, in short-term and medium-term ranges, to determine its ongoing effects on the target audience (CDCa, n.d.).  Owen (2007) also gives alternative usages for impact evaluation, suggesting it is used “to assess the effects of a settled program” (p.47), as well as echoing the CDC, that impact evaluation can also be used at intervals, such as a mid-point evaluation.  Khandker, Koolwal and Samad summarize impact evaluation as an assessment means to “help policy makers decide whether programs are generating intended effects; to promote accountability in the allocation of resources across public programs; and to fill gaps in understanding what works, what does not, and how measured changes in well-being are attributable to a particular project or policy intervention” (2010, p.3), which combines aspects of the OECD and CDC definitions.

Outcome Evaluation

Outcome evaluation “measures effect and changes that result from the campaign. Investigates to what extent the communication program/intervention is achieving its outcomes in the target populations” (CDCb, n.d., p.2), which is a similar definition to impact evaluation, creating confusion between the two approaches.  One contrast between outcome and impact evaluation focuses on who benefits from the program being evaluated; outcome evaluation seeks to assess how individuals are transformed during and after completion of a program (McNamara, n.d.), in contrast to evaluating community based transformation during impact evaluation. There is notable exception to the individual (outcome) vice community (impact) evaluation description simplification; the Centers for Disease Control and Prevention describe outcome evaluation as targeting the affects of programming on a specific population (CDCa).

This ambiguity in definition and usage makes it difficult to determine how best to utilize impact and outcome evaluation in professional practice. Which approach is best used to determine overall outcomes (transformations) of a program on a specific community? Alternatively, is it more pertinent to use impact evaluation to determine how well each component of a program is functioning with respect to meeting program goals?  How does the practitioner determine which evaluation approach is the best fit for each program to be evaluated?

References

Centers for Disease Control and Prevention [CDCa]. (n.d.). Types of evaluation [Web page].  Retrieved from: https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf

Centers for Disease Control and Prevention [CDCb]. Building Our Understanding: Key Concepts of Evaluation, What is it and how do you do it? Retrieved from: https://www.cdc.gov/nccdphp/dch/programs/healthycommunitiesprogram/tools/pdf/eval_planning.pdf

Khander, S.R., Koolwal, G.B., and Samad, H.A. (2010). Handbook on impact evaluation: Quantitative methods and practices. Washington, D.C.: The World Bank.

McNamara, C. (n.d). Basic guide to outcomes-based evaluation for nonprofit organizations with very limited resources [Web page]. Retrieved from: http://managementhelp.org/evaluation/outcomes-evaluation-guide.htm

Organisation for Economic Co-operation and Development [OECD]. Outline of principles of impact evaluation [Web page]. Retrieved from:http://www.oecd.org/dac/evaluation/dcdndep/37671602.pdf