What are the relative pros and cons of the situation judgment test as a predictor of manager performance?

Question

What will be an ideal response?

Answer 1

SJTs are often inexpensive to develop, administer, and score compared to other types of work samples described in this chapter (Clevenger, Pereira, Wiechmann, Schmitt, & Harvey, 2001). Also, the availability of new technology has made it possible to create and administer video-based SJTs effectively (Weekley & Jones, 1997). Regarding SJT validity, a meta-analysis based on 102 validity coefficients and 10,640 individuals found an average validity of .34 (without correcting for range restriction), and that validity was generalizable (McDaniel, Morgeson, Finnegan, Campion, & Braverman, 2001). Perhaps more important, SJTs have been shown to add incremental validity to the prediction of job performance above and beyond job knowledge, cognitive ability, job experience, the Big Five personality traits, and a composite score including cognitive ability and the Big-Five traits (Clevenger et al., 2001; McDaniel, Hartman, Whetzel, & Grubb, 2007; O'Connell, Hartman, McDaniel, Grubb, & Lawrence, 2007). SJTs also show less adverse impact based on ethnicity than do general cognitive ability tests (McDaniel & Nguyen, 2001). However, there are race-based differences favoring White compared to African American, Latino, and Asian American test takers, particularly when the instructions for taking the judgment test are g loaded (i.e., heavily influenced by general mental abilities) (Whetzel, McDaniel, & Nguyen, 2008). Thus, using a video-based SJT, which is not as heavily g loaded as a written SJT, seems like a very promising alternative given that a study found that the video format had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version (Lievens & Sackett, 2006).
In spite of these positive features, there are several challenges in using SJTs (McDaniel & Nguyen, 2001). Most notably, SJTs include a combination of different constructs and, while SJTs do work, we often do not understand why. Campion, Ployhart, and MacKenzie (2014) conducted a content analysis of 59 SJT articles published since 1990 and found that the most frequently assessed constructs are “applied social skills” (32.1%, n = 25), “heterogeneous composites” (24.4%, n = 19), and “leadership” (19.2%, n = 15). On average, SJTs included a mean of about six dimensions, many of which were very broad in nature and not defined very clearly or specifically, and they were measured using about 29 items.
The lack of clarity regarding constructs underlying SJTs was shown empirically by a reliability generalization study based on 98 articles that reported the internal reliability coefficient alpha of only .68 (Kasten & Freund, 2016). However, if SJTs are supposed to measure several dimensions, we should not expect high internal consistency reliability estimates (Sorrel et al., 2016). Nevertheless, greater clarity regarding which constructs are measured would result in greater understanding of reliability and its assessment.