Describe four types of comparison group designs that can be used for impact evaluation with social programs.

Question

What will be an ideal response?

Answer 1

1. Naïve estimates of program effects: A naïve effect estimate is what results when the average outcome for a group that participated in or had access to a program is simply compared with the average for another group that did not participate in the program or have access to it.
2. Covariate-adjusted, regression-based estimates of program effects: In one of the most common comparison group designs used by evaluators, outcomes for a group exposed to a program are compared with those for a comparison group selected on the basis of relevance and convenience. But in contrast to the naïve design, this design uses statistical techniques to adjust for differences between the groups that might bias the effect estimates. The first step in this approach is to measure a set of pre-intervention baseline characteristics for all the members of the study sample, focusing especially on characteristics expected to be related to the outcomes of interest. In this context, these variables are generally referred to as covariates. Those covariates are then used in a statistical prediction model that estimates the independent relationship of each covariate to a target outcome variable; that is, what each covariate contributes to the prediction of the outcome above and beyond the contributions of the other covariates included in the statistical model.
3. Matching designs, including propensity score matching: Another procedure for reducing bias in comparison group designs is matching. In a matched comparison, the intervention group is typically specified first and the evaluator then constructs a comparison group by selecting units unexposed to the intervention that match those in the intervention group on selected characteristics.
4. Interrupted time series designs: While those designs compared the outcomes of two groups—a program group and a comparison group—interrupted time series designs compare outcomes for a period before program implementation or participation with those observed afterwards. The program or other intervention in these designs “interrupts” a time series of periodic measures of a relevant outcome the program is expected to affect.