NN39, October 2007
Best Practice in Education and Training: Hype or Hope?
Impact Evaluations: The Ultimate Low Hurdle For ?Best Practices?
By David Ellerman, University of California at Riverside
KeywordsImpact evaluation, World Bank, Best practice
Summary
This article takes a critical look at the ?best practice? of ?impact evaluation? promoted by the World Bank.
----
There is an old critique of evaluations in the field of development assistance. Basically, the critique is that it is well-nigh impossible to get an evaluation that is truly independent. Usually those who fund an evaluation are closely related to the project and will not welcome bad news as it reflects on their judgment and expertise. This lesson is not lost on the evaluators who draw their income from this business. Hence, aside from minor wrist-slapping to show ?independence,? evaluators are rarely the bearers of bad news, and project administrators are duly thankful for the ?good job? the evaluators are doing.
But recently the problems with development evaluation have been carried to a new ?high? by the most recent fad of ?impact evaluations.? The root function of an evaluation is, as the name indicates, to judge the value of something. But the value of something is only determined by considering its costs which entails comparing it to the alternatives. One of the most basic concepts in economics is the notion of opportunity costs. When certain resources are devoted to Plan A, what is the real cost of the resource commitment? The resources might include time and effort and many other difficult-to-quantify investments so it is not just a matter of adding up the costs. The doctrine of opportunity cost says that the true cost of committing those resources to Plan A is the benefit that was foregone by not committing those resources to the best alternative Plan B. Thus the notion of evaluation leads directly to the ?compared-to-what?? question. If the benefit from investing those resources in Plan A was greater than the opportunity cost (i.e., the benefit from the best alternative Plan B using those resources), then Plan A has a positive net value and gets a positive evaluation.
This basic concept of evaluation is not ?rocket science??in fact it is just elementary economics. Yet, we now have the fad of ?impact evaluation? in full flower and it is promoted by many economists who should know better. The basic idea of an ?impact evaluation? of a plan or project is not to compare it to alternatives using comparable resources but to compare it to the alternative of doing nothing. In other words, the ?counterfactual? is not the best alternative use of the same resources in the current circumstances but what would have happened if no resources had been expended in the current circumstances. No matter how what resources are expended, if the results are ?better than nothing? then the project gets a ?positive evaluation.?
When described in such stark terms, one might wonder how ?impact evaluation? could be seriously promoted. One treatise devoted to the subject notes on the first page:
Notice that in concentrating on impact analysis we will not be concerned in this book with the worthwhileness of a programme, as in benefit-cost analysis, for example, but rather, we will limit our concern to certain of its accomplishments. [Mohr, Lawrence B. 1988. Impact Analysis for Program Evaluation. Chicago: Dorsey Press, p. 1]
How can a treatise on ?programme evaluation? not consider the ?worthwhileness? of a programme? Apparently, it is enough to say that it is an ?impact evaluation? or an ?impact analysis.? Inside many aid organizations concerned to justify their programmes, the tell-tale adjective ?impact? is soon dropped so that we have so-called ?evaluations? that do not consider the worthwhileness of the programmes.
In this as in so many other aspects of development assistance, the World Bank has been the leading opinion-maker and practitioner. Indeed, the World Bank's Development Economics Department (i.e., the Bank?s research department) has launched a major initiative to promote impact evaluations themselves as the ultimate low-hurdle way to evaluate development programmes and verify best practices.
Impact evaluation is an assessment of the extent to which interventions or programmes cause changes in the well-being of target populations, such as individuals, households, organizations, communities, or other identifiable units to which interventions are directed in social programmes. One way of conceptualizing net effects (or outcome) is the difference between persons or other targets that have participated in a project and comparable individuals, or entities that have not participated in the project.
An impact evaluation must estimate the counterfactual, which attempts to define a hypothetical situation that would occur in the absence of the programme, and to measure the welfare levels of individuals or other identifiable units that correspond with this hypothetical situation. [1]
Perhaps there is some irony here. The World Bank often tries to legitimate its leading role by citing its unique standpoint to scan the whole world for alternatives and to ascertain ?best practices.? Yet after decades of failures, it has now decided that the best way to evaluate its development programmes is not to compare them to all the actual alternatives that might be undertaken with the same considerable resources but to compare them to ?a hypothetical situation that would occur in the absence of the programme.?
A genuine evaluation not only considers costs and real alternatives but should be seen as an integral part of the process of social learning. Under conditions of uncertainty, local variation, and an acknowledged Socratic ignorance of ?The Solution,? the best approach to social learning seems to be parallel experimentation and the real-time evaluation of benchmarking and communication of ideas between actual (i.e., non-hypothetical) experiments where comparable resources were expended as opposed to no resources [see Ellerman, David 2005. Helping People Help Themselves: From the World Bank to an Alternative Philosophy of Development Assistance. Ann Arbor: University of Michigan Press].
Notes
[1] On the World Bank?s website, www.worldbank.org, click on "Data & Research", then "Impact Evaluation", and then "Overview" to find this description and to find a handbook [Baker, Judy L. 2000. Evaluating the Impact of Development Programmes on Poverty: A Handbook for Practitioners. Washington DC: World Bank].
Back to full contents of NORRAG NEWS 39.
Download the full issue of NORRAG NEWS 39 in pdf.