Kapp and Anderson discuss 10 classic threats to internal validity. Discuss and explain 5 of these threats, and give an example of each.

Question

What will be an ideal response?

Answer 1

The 10 threats to internal validity are: (1) History: This involves contextual and outside events (other than the independent variable) that coincide with the introduction of the independent variable or occur during data collection that could have affected the participant’s experience and the program’s outcome. If discovered during the course of the evaluation, these historical events should be identified and addressed to the extent possible. For example, the increase of sorrow or hopelessness in the sample may be a result of certain local or national events rather than a failure of the depression-reducing intervention. (2) Maturation and the passage of time: This involves natural changes (physical growth, mental growth, emotional maturation, and motivation) that take place and affect the behaviors, feelings, thoughts, and responses of participants in a program evaluation. For example, the increasing intelligence or coordination of school-aged children may be less of an outcome from an intervention and more attributable to normal cognitive and physical development. (3) Testing: The initial measurement of one’s actions or thoughts or a test for a certain quality may influence the person’s behavior and consequently affect the outcome of the post-test. The very process of testing may enhance performance on a test. For example, a person may work very hard to improve his or her test score; it is this motivation and test-taking experience that makes the difference, not the program. (4) Instrumentation: The survey, questionnaire, observational guide, or other measuring instrument used to collect data may change between the beginning of the study and the study’s conclusion. For example, an evaluator may find that several questions that would be helpful to ask should be added to a questionnaire, or questions that are persistently confusing should be reworded, changing the content of the questionnaire that has already been administered to others in the sample. (5) Statistical regression: The results that report significant change -improvement or deterioration-may be the result of having measured an extreme behavior or an extreme position. For example, there may be “nowhere to go but up” as a person is at his or her lowest point of functioning or mood; or there is “nowhere to go but down” as the person cannot be reasonably expected to maintain or repeat an extraordinarily effective performance. (6) Selection bias: The manner in which persons are selected for the study and the type of persons selected affects their reaction to the intervention. For example, choosing new employees in a study on agency climate or conducting exit interviews with persons who have been fired will affect the information and viewpoints gathered in an evaluation intended to give a representative view of the agency. (7) Experimental mortality and attrition: The composition and complexion of a sample may change as persons drop out, as intended participants fail to respond, or as subjects relocate during the conduct of the evaluation. The study may turn out to be a de facto survey of program survivors rather than involving a full range of recipients of services. (8) Ambiguity about the direction of causal influences: The direction of the impacts and influencing conditions may not be clear or discernible. For example, are people depressed because they are not getting sufficient sleep, or are they not getting sufficient sleep because they are depressed? (9) Design contamination: The persons who are participating in a study may behave differently because they know they are being studied. This can result in heightened success or failure for the intervention. This contamination is also called reactive effects. For example, workers know that their use of time during the workday is part of the evaluation; therefore, with greater attentiveness to their use of time, they order their days in a manner that will reflect most positively on them. (10) Diffusion or imitation of treatments: The study is intending to examine the unique qualities and approach associated with an innovative program. However, many aspects of the approach, philosophy, and techniques associated with the intervention have already influenced the larger setting that might have provided a basis for comparison. For example, comparing a strength-based intervention with traditional practice may not show dramatic results as strength-based language, techniques, and values may have been adopted by or influenced traditional practice.