Describe how we collect, analyze, and interpret data for a quantitative item analysis. Give examples.

Question

What will be an ideal response?

Answer 1

• Item difficulty:
o Items that everyone gets “right” or everyone gets “wrong” provide no basis for comparison and yield similar test scores for all test takers. Therefore, developers analyze each test item for its difficulty.
o We define item difficulty as the percentage of test takers who respond correctly. Items with difficulty levels or p values of .5 yield distributions of test scores with the most variation.

• Because difficulty levels can be expected to vary, most developers seek a range of difficulty levels among all items that averages approximately .5. They discard or rewrite items with extreme p values, usually defined as 0 to .2 (too difficult) and .9 to 1 (too easy).

• Item discrimination:
o Because inferences regarding the meaning of high test scores versus low test scores will be made by the users of a test, it is important to obtain a measure of how well each item separates those test takers who demonstrate a high degree of skill, knowledge, attitude, or personality characteristic from those who demonstrate little of the same skill, knowledge, attitude, or personality characteristic.
o One of the ways that test developers evaluate this is by calculating a discrimination index, which compares the performance of those who obtained very high test scores with the performance of those who obtained very low test scores on each item.
o Low positive numbers suggest that nearly as many people who had low scores responded correctly, as did those who had high scores. Each of these situations indicates that the item is not discriminating between high scorers and low scorers. Therefore, test developers discard or rewrite items that have low or negative D values.

• Interitem correlation matrix:
o Another important step in the item analysis is the construction of an interitem correlation matrix, which displays the correlation of each item with every other item.
o These coefficients are interpreted like Pearson product–moment correlation coefficients.
o The interitem correlation matrix provides important information for increasing the test’s internal consistency. Ideally, each item should be correlated with every other item measuring the same construct and should not be correlated with items measuring a different construct.

• Item total correlations:
o Another way to assess the contribution of a single item to overall test consistency as well as assess the item’s ability to discriminate high-scoring individuals from lower- scoring individuals is to calculate the item–total correlation.
o This is a measure of the strength and direction of the relation between the way test takers responded to one item and the way they responded to all of the items as a whole.
o Testing practitioners use item–total correlations as an alternative to the discrimination index discussed earlier. When items on a test are all measuring a similar construct, it is reasonable to assume that, on average, individuals who answer a particular question correctly will obtain a higher total score on the test than will individuals who answer the question incorrectly.

• Item criterion correlation:
o Some developers also correlate item responses with a criterion measure such as a measure of job performance for pre-employment tests or a diagnostic measure for clinical tests.

• Item characteristic curve (ICC):
o When IRT is used to conduct an item analysis, ICCs will be reviewed.
o An ICC is the line that results when we graph the probability of answering an item correctly with the level of ability on the construct being measured. The ICC provides a picture of the item’s difficulty and how well it discriminates high performers from low performers.