Why is a true experimental design the strongest method for conducting impact evaluations?

Question

What will be an ideal response?

Answer 1

The Risk Skills Training Program (RSTP) was designed to target multiple risk behaviors such as drinking and drug use along with changing adolescents’ personal beliefs about such behavior. Elizabeth D’Amico and Kim Fromme (2002) studied the impact of the RSTP and how it compared to the Drug Abuse Resistance Education-Abbreviated (DARE-A) program as well as how both of these programs affected outcomes compared to students who received no programming (a control group).
To evaluate the efficacy of these programs in reducing drinking and drug use, they randomly selected 150 students to participate in their study. Then students were randomly assigned to one of the three conditions: 75 students received RSTP programming, 75 students received the DARE-A programming, and another 150 students were randomly selected to participate but received no programming. The students received a pretest assessment and then posttest assessments, which took place at both two and six months after the programs.
The impacts (dependent variables) D’Amico and Fromme (2002) examined included positive and negative “alcohol expectancies” (the anticipated effects of drinking) as well as perceptions of peer risk taking and actual alcohol consumption. D’Amico and Fromme found that negative alcohol expectancies increased for the RSTP group in the posttest but not for the DARE-A group or the control group, while weekly drinking and “positive expectancies” for drinking outcomes actually increased for the DARE-A group and/or the control group by the six-month follow-up but not for the RSTP group (see Exhibit 10.3).
You should recognize the design used by D’Amico and Fromme as a true experimental design (see Chapter 6). This is the preferred method for maximizing internal validity—that is, for making sure your causal claims about program impact are justified. Cases are assigned randomly to one or more experimental treatment groups and to a control group so that there is no systematic difference between the groups at the outset. The goal is to achieve a fair, unbiased test of the program itself so that the judgment about the program’s impact is not influenced by differences between the types of people who are in the different groups. It can be a difficult goal to achieve, because the usual practice in social programs is to let people decide for themselves whether they want to enter a program and also to establish eligibility criteria that ensure that people who enter the program are different from those who do not (Boruch, 1997). In either case, a selection bias is introduced.
Impact analysis is an important undertaking that fully deserves the attention it has been given in government program funding requirements. However, you should realize that more rigorous evaluation designs are less likely to conclude that a program has the desired effect; as the standard of proof goes up, success is harder to demonstrate.