Describe commonly used experimental and quasi-experimental designs. What potential threats to validity must be guarded against for each of these designs?

Question

What will be an ideal response?

Answer 1

There are six commonly used quasi-experimental designs and four commonly used experimental designs. The first quasi-experimental design is the single group case study, in which the experiment or intervention is represented by X, and O1 stands for the outcome measurement of the experiment or intervention. Time moves from phase 1 (pre-intervention) to phase 2 (post-intervention).


The fatal flaw of this design is that there is no baseline measurement to provide comparison with the intervention outcome, nor is there any comparison group. Thus, it is impossible to tell whether there is an improvement in outcome. Other threats to the internal validity of this simple case study design are attrition, maturation, and history.

A second quasi-experimental design, the single group before-and-after design, consists of measuring or making observations of one group of subjects (the pre-test), implementing an intervention or program (the independent variable), and measuring the subjects at the end of the intervention or program (the post-test). The pre-intervention observations are represented by O1, the independent variable by X, and the post-intervention observations by O2.


The threats to the validity of this design include maturation, history, testing, instrumentation, and statistical regression. Unless objective reference or norm groups are available for comparison, considerable problems will occur when researchers try to make any judgments about program X on the basis of this design.

The third quasi-experimental design is the comparison group post-test only. Each row represents a separate group, the intervention or program is represented by X, and the blank space under X stands for the no-intervention comparison group. The outcome of the dependent variable is measured by O, and each group is measured just once (i.e., post-intervention).


The most serious threat to the validity of this design is selection bias because, without the random assignment of subjects to the study and comparison groups, there is no control of possible pre-intervention differences. Other threats include attrition, which is uncontrolled because no pre-test data exist by which the investigator may find out whether subjects who drop out of a study are comparable to those who remain; maturation, which may become a problem if the study is of long duration and if maturational factors are operating differently in the two groups; and contamination, if the study group interacts with the comparison group and shares information that may change outcomes.

The fourth quasi-experimental design is the nonequivalent comparison group pre-test–post-test, in which measurements are made in both study and comparison groups prior to intervention.

If the groups in this design are similar, then the design controls for history, maturation, testing, and regression to the mean. However, contamination and attrition (particularly differential attrition rates between study and comparison groups) could be potential biases to this design.

The fifth quasi-experimental design is the single group time series, which uses the subjects in the experimental intervention or program as their own control group. The same measurement is made on intervention or program subjects at regular intervals several times before and after the intervention.

History may represent a potential threat to internal validity in this design. Even if a clear change is noted in the observations following the implementation of program X, it is difficult to know for certain whether X caused the change or whether X just happened to occur at about the time that the measures would have changed anyway because of some other outside events. Testing may also pose a threat to internal validity because individuals with repeated measurements may improve their scores due to practice rather than the intervention.

The last quasi-experimental design is the time series with a nonequivalent control group. It is like the single group time series except with the addition of a nonequivalent control group, and it also incorporates the pre-test–post-test nonequivalent control group design. Two groups of subjects are measured regularly both before and after the intervention or program, which is available to one group but not the other.


The addition of a comparison group makes it possible to control for external events that might take place during the course of the study, and the alternative explanation for the results due to external influence or history can often be ruled out. In addition, the selection–maturation interaction is controlled to the extent that, if the study group generally showed a greater rate of gain, it would be seen in the pre-intervention observations as well.

The single most important difference between experimental and quasi-experimental designs is randomization, or the random assignment of subjects to experimental or control groups. Randomization is present in experimental designs, but absent in quasi-experimental designs. Randomization is represented by the letter R, and the other symbols are joined by R, with the study period shown as times moves from left to right. The rows represent study or experimental and control groups. Observation or measurement of the dependent variable is indicated by O, and the intervention or program of the independent variable is represented by X.

The first experimental design is the true control group pre-test–post-test, which is like the nonequivalent comparison group pre-test–post-test, except that the control group is formed by random assignment. Research subjects are randomly assigned to either the experimental or control groups. Subjects assigned to the experimental group receive experimental intervention or program X. Subjects assigned to the control group do not receive experimental intervention or program X, but may get an alternative program (X). The pre-test–post-test control group design involves measuring both the experimental and control groups at approximately the same time before and after the intervention or program. The pre-test results can be used to assess the similarities or equivalencies between the two groups. The post-test results are used to assess the impact of the intervention or program (X).


This design controls the threats to internal validity, but is still susceptible to threats to external validity, which reflects the extent to which the experimental results may be generalized. Attrition and contamination remain potential threats.

The second experimental design is the true control group post-test-only, in which subjects are randomly assigned to experimental or control groups. No pre-test or measurement is given. Instead, the intervention or program is administered to the experimental group, while the control group receives no intervention. Measurement is made after the intervention for both groups.


This design controls for the common threats to internal validity adequately.

The third experimental design is the Solomon four-group, which is relatively complex and integrates both the true control group pre-test–post-test and the true control group post-test-only. There are two experimental groups and two control groups. A randomization procedure is used in assigning subjects to each of the four groups. Pre-test measurement is used for one pair of experimental control groups, but not for the other. The same intervention is implemented in both groups, and post-test measurement is used for all four groups at about the same time. This design combines the strengths of the previous two experimental designs and enables the researcher to make additional comparisons.


This design effectively controls for threats to internal validity, but may be twice as expensive due to the addition of two extra groups.

The final experimental design is the time series with an equivalent control group, and is like the time series with a nonequivalent control group except with the addition of random assignment. Two groups of subjects are randomly selected and measured regularly before the program, and then only one group gets program X. The other might get an alternative program or no program.
This is a very powerful design effectively controlling for all threats to internal validity, but it is also very expensive and is seldom used. The main threats to validity come from external validity threats.