Explain how bias can be introduced in impact evaluation and provide examples of different types of bias.

Question

What will be an ideal response?

Answer 1

Bias is present when either the measurement of the outcome with program exposure or the estimate of the counterfactual outcome departs from the corresponding true value. Unfortunately for the evaluator and other stakeholders, the extent of the bias cannot be determined from the data collected for an impact evaluation, leaving some degree of uncertainty about the validity of the effect estimates with even the strongest of these designs. Selection bias arises when some process that is not fully known influenceswhether individuals enter into the program group or the comparison group with no assurance that this process selected completely comparable individuals for each group. Naturally occurring trends in the community, region, or country, sometimes termed secular drift, may produce changes that enhance or mask actualprogram effects. Sometimes,short-term events can produce changes that distort the estimates of program effect. Impact evaluations must often cope with natural maturational and developmental processes that can produce change in a study population independent of program effects. If those changes affect one group in a comparison group design more than the other, they will bias the program effect estimates.Another potential source of bias is associated with the tendency for more extreme outcomes to naturally drift in a less extreme direction over subsequent time periods.