Discuss and give examples of the five sources of evidence of validity described in the newest Standards for Educational and Psychological Testing.

Question

What will be an ideal response?

Answer 1

• Evidence based on test content: This source of validity evidence involves logically examining and evaluating the content of a test (including the test questions, format, wording, and tasks required of test takers) to determine the extent to which the content is representative of the concepts that the test is designed to measure without either underrepresenting those concepts or including elements that are irrelevant to their measurement.

• Evidence based on response processes: This source of validity evidence involves observing test takers as they respond to the test and/or interviewing them when they complete the test. We use these observations and interviews to understand the mental processes that test takers use to respond. For instance, if the test was designed to measure logical reasoning, test takers should report mentally processing the test information while they solve the test problems. If the test is scored by using trained observers or judges, this source of validity evidence also involves exploring whether those observers or judges used the criteria that were defined to document and evaluate test taker behaviors or performances.

• Evidence based on internal structure: This type focuses on whether the conceptual framework used in test development could be demonstrated using appropriate analytical techniques. For instance, if a test was designed to measure a single concept (such as anxiety), we would analyze the test results to find out how many underlying concepts account for the variations in test taker scores. If the test was designed to measure one concept only, the analysis should show that only one concept accounts for a majority of the information the test takers provided. If the analysis suggests that the scores were affected by more than one underlying concept or factor, then we would question whether we had evidence of validity of the test based on its underlying single concept structure.

• Evidence based on relations with other variables: This source of validity evidence typically involves correlating test scores with other measures to determine whether those scores are related to other measures to which we would expect them to relate. Likewise, we would want to know that the test scores are not related to other measures to which we would not expect them to relate. For instance, a test designed to be used in employee selection should correlate with measures of job performance. A test of mechanical aptitude should correlate with another test of mechanical aptitude, while not correlating with a vocabulary test.

• Evidence based on the consequences of testing: Any time we make a psychological measurement, both intended and unintended consequences may occur. For example, an intended consequence of a test for personnel selection would be obtaining accurate information for hiring. However, if the test is biased, an unintended consequence might be that test scores appear to favor one group over another.