While most project or programme evaluations focus on whether implementation of operations or provision of. By using impact evaluations, we know how effective our programs are and can make good, costeffective decisions about future programs to. Impact evaluation using differenceindifferences emerald insight. Overview of impact evaluation page 2 an impact evaluation should only be undertaken when its intended use can be clearly identified and when it is likely to be able to produce useful findings, taking into account the availability of resources and the timing of decisions about the programme or policy under investigation. Differenceindifferences is one of the most widely applied methods for estimating causal effects of programs when the program was not implemented as. The three impact evaluation methods discussed up to this pointrandomized. This provides a highlevel overview of the steps involved in the evaluation of road safety programmes, from defining the problem to reporting results. Evaluation evaluation is the periodic analysis of data and information, as distinct from monitoring, to learn, improve and assess performance. Our attribution strategy for measuring the impact of the tv intervention was to use the difference in difference method did. Impact evaluation using differenceindifferences emerald.
The differenceindifference did technique originated in the field of econometrics, but the logic underlying the technique has been used as early as the 1850s by john snow and is called the controlled beforeandafter study in some social sciences. Difference model lets think about a simple evaluation of a policy. Impact heterogeneity the variation in impact as a result of differences in context, beneficiary characteristic or implementation of the intervention. Needs assessment process evaluation impact evaluation to address the low literacy levels, the government will deliver free text books to schools to improve passing rates and increase employment prospects of school leavers. Impact evaluation using differenceindifferences scielo. The us nhtsa 2 has developed a guide to evaluating road safety projects. Jul 01, 2016 due to the limitations of psm and iptw, the difference in differences did method may be an alternative methodology. Impact evaluation a study of the attribution of changes in the outcome to the intervention. This allows for correction for any differences between the treatment and comparison groups that remain constant over time. Pdf impact evaluation using differenceindifferences. Impact evaluation methodologies impact evaluation workshop, lilongwe, dec. Monitoring vs evaluation between monitoring and evaluation of projects, one can find a variety of differences. Monitoring and evaluation are two states of analysis in terms of the progress made in relation to the goals of an enterprise or a firm. Choosing an impact evaluation method 187 determining which method to use for a given program 187 how a programs rules of operation can help choose an impact evaluation method 188 a comparison of impact evaluation methods 193 finding the smallest feasible unit of intervention 197 chapter 12.
If there is no relationship between policy implementation and subsequent outcomes, then the differenceindifferences estimate is equal to 0 figure, a. Global developments evaluation gap working group in 2004, international initiative on impact evaluation 3ie in 2008, masters of science in impact evaluation for international development at the university of east anglia, and the centre for development impact at the institute of development studies, among others. Section v discusses issues of operational implication including choosing an evaluation method, designing steps, and resource commitments. Types and uses of evaluation in order to plan the evaluation in accord with the most appropriate evaluation method, it is necessary to understand the difference between evaluation types. An intuitive introduction to differenceindifferences youtube.
Using matching techniques to select participant and nonparticipant households in the health insurance subsidy program 111 the matching method at work 1. Differenceindifference estimation columbia university mailman. Designmethodologyapproach the paper describes the did method, starting with an intuitive explanation, goes through the main assumptions. Differenceindifferences method in comparative effectiveness. If we have data on a bunch of people right before the policy is enacted and on the same group of people after it is enacted we. Differenceindifference estimation columbia university. Impact evaluations need to account for the counterfactual what would have occurred without the intervention through the use of an experimental or quasiexperimental design using comparison and treatment groups. A guide to statistical methods for program impact evaluation. Limitations of the difference in differences method 104 notes 104 references 105. Difference between monitoring and evaluation compare the. Hence, differenceindifference is a useful technique to use when randomization on the individual level is not possible. Evaluation constitutes part of an ongoing cycle of. Differenceindifferences did is one of the most frequently used methods in impact evaluation studies.
Difference in differences did is one of the most frequently used methods in impact evaluation studies. Evaluation results can be used to maintain or improve program quality and to ensure that future planning can be more evidencebased. Pdf programme evaluation using difference in difference. An intuitive introduction to differenceindifferences. Explains that the differenceindifferences dd method compares the changes. Using the difference in difference method for impact assessments. Review of the basic methodology since the work by ashenfelter and card 1985, the use of difference in differences methods has become very widespread. What is the difference between an impact evaluation and a project evaluation. Synthetic control method, difference in differences, matching, policy evaluation, payforperformance introduction natural experiments can exploit exogenous variation across time periods and geographical areas to identify the causal effects of alternative policies jones and rice 2011. Schooling and labor market consequences of school construction in indonesia. Difference in difference may be thought as a method that tries to improve upon the ba method 26.
University of california, san diego march 24th, 2010. Impact evaluations have either an experimental or quasiexperimental design. Explains that the differenceindifferences dd method compares the changes in outcomes over time between a population enrolled in a program the treatment. The differenceindifference did technique originated in the field of econometrics, but the logic.
Based on a combination of beforeafter and treatmentcontrol group comparisons, the method has an intuitive appeal and has been widely used in economics, public policy, health research, management and other fields. Quantitative and qualitative methods in impact evaluation and. Historically, did has been used in the evaluation of healthcare policy, as it allows the researcher to control for background changes in outcomes. For example, the findings of an impact evaluation can be used to improve implementation of a programme for the next intake of participants by identifying critical elements to monitor and tightly manage. There are a variety of evaluation designs, and the type of evaluation should match the development level of the program or program activity appropriately. Jan 29, 2015 difference in differences is one of the most widely applied methods for estimating causal effects of programs when the program was not implemented as a randomized controlled trial. This paper does not intend to provide a detailed account of different approaches to impact evaluation nor an overview of proposed solutions to specific impact evaluation challenges2. Although evaluation designs may produce useful information about a programs effectiveness, some may produce more useful information than others. These two states of analysis differ in their manner of approach. Research on the causal effects of the treatment condition revolves around the. Material in this presentation developed from cega and world bank materials. Most often, impact evaluation is used for summative purposes. Introduction impact evaluation is increasingly recognized as an important part of the evaluation toolkit, resulting in calls for more and better impact evaluation.
In contrast, if the policy is associatedwithbeneficialchanges,thentheoutcomesfollowingimple. The challenge of using the difference in difference method for impact assessments. Explains that the differenceindifferences dd method compares the changes in outcomes over time between a population enrolled in a program the treatment group and a population not enrolled the comparison group. An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on causeandeffect analysis.
An earlier asian development bank publication adb 2006 provides an overview of impact evaluation, including a brief presentation of the methodology, two case studies, and discussions on practical issues related to carrying out impact evaluation. Evaluation is a systematic process to understand what a program does and how well the program does it. Operationalizing the impact evaluation design 143 choosing an impact evaluation method 143 is the evaluation ethical. Purpose of impact evaluation impact evaluation serves both objectives of evaluation. The key questions that the evaluation will want to address is.
Difindif removes the trend effect from the estimate of impact using the ba method. Guilkey, aiko hattori and gustavo angeles for measure evaluation, outlines core statistical and econometric methods for program impact evaluation and includes a section on using the difference in difference model. An impact evaluation measures the programs effects and the extent to which its goals were attained. Section vi introduces two examples of impact evaluation and explains in greater detail. The change in the control group provides an estimate of the trend. Review your logic model to think about each of your proposed outcomes or objectives, and impact. Did requires data from prepostintervention, such as cohort or panel data individual level data over time or repeated crosssectional data individual or group level. When does rigorous impact evaluation make a difference. The case of the millennium villages abstract when is the rigorous impact evaluation of development projects a luxury, and when a necessity. Whatever happened to the control group over time is what would have happened to the treatment group in the absence of the program. Outline of principles of impact evalation discussion dra. We first conducted a service provider level assessment of the tv channel programme staff to check whether they had incorporated elements. This measure can suffer from various problems, so that a double difference, comparing the difference in the change in the outcome for treatment and comparison groups, is to be preferred.