Describe the processes and expected outcomes of validation and cross-validation studies. Explain why each is important.

Question

What will be an ideal response?

Answer 1

• Standards for setting up the validation study are similar to those suggested for designing the pilot study.

• The validation study should take place in one or more situations that match the actual circumstances in which the test will be used.

• Using more than one test site will provide evidence that the results are generalizable, meaning the test can be expected to produce similar results even though it has been administered in different locations.

• Developers choose as test takers a sample of people who resemble or are part of the test’s target audience.

• The sample should be large enough to provide the power to conduct the desired statistical tests.

• Data can be collected on the test’s reliability/precision, its correlation with any appropriate
outside criteria such as performance evaluations, and its correlation with other measures of the test’s construct depending on the evidence that is appropriate to be collected for the particular test.

• If the final revision yields scores with sufficient evidence of reliability/precision and validity, the test developers conduct a final analysis called cross-validation for tests that rely on criterion-related evidence of validity to make predictions.

• The process of cross-validation involves a final round of test administration to another sample of test takers representative of the target audience.

• Usually the validity coefficient will be lower than the one found in the original validation study because of chance factors that contributed to the validity coefficient in the original study.

• The reduction the in cross-validated coefficient is called shrinkage.

• Cross validation can be conducted with a single group of test takers if it is large enough to break the group in two samples. The regression statistics that are derived from the first group are used to estimate the criterion in the second group. The better the match between the predicted criterion values in the second group with the actual values on the criterion, the lower the shrinkage.