Applied Psychology

Applied Psychology

Investigating The Psychometric Properties of The Situational Judgment Test for Assessing Employees' Job Competency

Document Type : Original Article

Authors
Abolfazl Ghadami 1
Enayatollah Zamanpour 2
Jalil Younesi 3
Mehdi Zarea Bahramabadi 4
1 Ph.D. Student, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.
2 Assistant Prof, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.
3 Associate Prof, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.
4 Associate Prof, Faculty of Counseling, Allameh Tabataba'i University, Tehran, Iran.
10.48308/apsy.2025.237987.1745
Abstract
Aim: This study aimed to assess the psychometric properties of a situational judgment test (SJT) designed to evaluate job-related competencies among employees.

Method: This applied, instrument-development study was conducted with 300 private-sector employees in Iran, selected via convenience sampling. A researcher-developed SJT assessed competencies like responsibility and communication. Analyses included Cronbach’s alpha for reliability, item difficulty/discrimination indices, and confirmatory factor analysis using SPSS-23, LISREL 8.8, and Jmetrik V4.0.5.

Results: The SJT demonstrated good reliability and validity. Normative scores were developed for the target population.

Conclusion: The SJT is a reliable and valid tool for identifying behavioral competencies and is recommended for primary and secondary screening in employee selection.
Keywords
Situational Judgment Test
Normalizing
Behavioral Competencies
Psychometric Properties
Subjects
Allen, J. G. (2004). The complete Q&A job interview book. John Wiley & Sons. [Link]
Bergman, M. E., Drasgow, F., Donovan, M. A., Henning, J. B., & Juraska, S. E. (2006). Scoring situational judgment tests: Once you get the data, your troubles begin. International Journal of Selection and Assessment, 14(3), 223-235. [Link]
Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62(2), 229-258. [Link]
Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023). Performance of ChatGPT on the situational judgement test—a professional dilemmas–based examination for doctors in the United Kingdom. JMIR Medical Education, 9(1), e48978. [Link]
Brady, M. (2019). A Situational Judgment Test of Self-Control and its Relationship to Academic Performance: Development of a New Measure. [Link]
Brooks, M. E., Highhouse, S., Weekley, J. A., & Ployhart, R. E. (2006). Can good judgment be measured? Situational judgment tests: Theory, measurement, and application, 39-55. [Link]
Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta‐analysis of their criterion‐related validities. Personnel Psychology, 63(1), 83-117. [Link]
Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86(3), 410. [Link]
Drucker, P. (2012). Managing in a time of great change. Routledge. [Link]
Flick, U. (2021). Doing interview research: The essential how to guide. [Link]
Freudenstein, J. P., Schäpers, P., Reznik, N., Stolte, T., & Krumm, S. (2024). The influence of situational strength on the relation of personality and situational judgment test performance. International Journal of Selection and Assessment, 32(1), 1-11. [Link]
Golubovich, J., Lake, C. J., Anguiano-Carrasco, C., & Seybert, J. (2020). Measuring achievement striving via a situational judgment test: The value of additional context. Revista de Psicología Del Trabajo y de las Organizaciones, 36(2), 157-168. [Link]
Golubovich, J., Seybert, J., Martin-Raugh, M., Naemi, B., Vega, R. P., & Roberts, R. D. (2017). Assessing perceptions of interpersonal behavior with a video-based situational judgment test. International Journal of Testing, 17(3), 191-209. [Link]
Grossman, G., & Sharf, R. (2018). Situational Judgment Tests and Transformational Leadership: An Examination of the Decisions, Leadership, and Experience in Undergraduate Leadership Development. Journal of Leadership Education, 17(1). [Link]
Harwood, H., Roulin, N., & Iqbal, M. Z. (2024). “Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission. Journal of Vocational Behavior, 154, 104013. [Link]
Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical assessment measures. Psychological assessment, 15(4), 456. [Link]
Hedlund, J., Forsythe, G. B., Horvath, J. A., Williams, W. M., Snook, S., & Sternberg, R. J. (2003). Identifying and assessing tacit knowledge: Understanding the practical intelligence of military leaders. The leadership quarterly, 14(2), 117-140. [Link]
Heininger, S. K., Baumgartner, M., Zehner, F., Burgkart, R., Söllner, N., Berberat, P. O., & Gartmeier, M. (2021). Measuring hygiene competence: the picture-based situational judgement test HygiKo. BMC medical education, 21(1), 410. [Link]
Hernandez, J., Muratet, M., Pierotti, M., & Carron, T. (2022, July). Enhancement of a Gamified Situational Judgment Test Scoring System for Behavioral Assessment. In 2022 International Conference on Advanced Learning Technologies (ICALT) (pp. 374-378). IEEE. [Link]
Hickman, L., Bosch, N., Ng, V., Saef, R., Tay, L., & Woo, S. E. (2022). Automated video interview personality assessments: Reliability, validity, and generalizability investigations. Journal of Applied Psychology, 107(8), 1323. [Link]
Hoffman, B. J., Kennedy, C. L., LoPilato, A. C., Monahan, E. L., & Lance, C. E. (2015). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology, 100(4), 1143. [Link]
Jesiek, B. K., Woo, S. E., Parrigon, S., & Porter, C. M. (2020). Development of a situational judgment test for global engineering competency. Journal of Engineering Education, 109(3), 470-490. [Link]
Khosrojedi, Z., & Heidari, M. (2024). Research Article The Psychometric Characteristics of Strength‐Based Parenting Scale, Parent Child Version. Applied Psychology, 18(2), 86-106. [Link]
Kumazaki, H., Yoshikawa, Y., Muramatsu, T., Haraguchi, H., Fujisato, H., Sakai, K., ... & Mimura, M. (2021). Group-Based Online Job Interview Training Program Using Virtual Robot for Individuals With Autism Spectrum Disorders. Frontiers in psychiatry, 12. [Link]
Landers, R. N., Auer, E. M., & Abraham, J. D. (2020). Gamifying a situational judgment test with immersion and control game elements: Effects on applicant reactions and construct validity. Journal of Managerial Psychology, 35(4), 225-239. [Link]
Langer, M., König, C. J., & Papathanasiou, M. (2019). Highly automated job interviews: Acceptance under the influence of stakes. International Journal of Selection and Assessment, 27(3), 217-234. [Link]
Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: a comparison in terms of predictive validity. Journal of applied psychology, 91(5), 1181. [Link]
Lievens, F., & Sackett, P. R. (2012). The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. Journal of Applied Psychology, 97(2), 460. [Link]
Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426-441. [Link]
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & GRUBB III, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta‐analysis. Personnel psychology, 60(1), 63-91. [Link]
McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: a clarification of the literature. Journal of Applied Psychology, 86(4), 730. [Link]
Meyer, R. D., Dalal, R. S., & Hermida, R. (2010). A review and synthesis of situational strength in the organizational sciences. Journal of management, 36(1), 121-140. [Link]
Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (2006). An alternative selection procedure: The low-fidelity simulation. Journal of applied psychology, 75(6), 640. [Link]
Murano, D., Lipnevich, A. A., Walton, K. E., Burrus, J., Way, J. D., & Anguiano-Carrasco, C. (2021). Measuring social and emotional skills in elementary students: Development of self-report Likert, situational judgment test, and forced choice items. Personality and Individual Differences, 169, 110012. [Link]
Nørskov, S., & Ulhøi, J. P. (2020). The use of robots in job interviews. Encyclopedia of electronic HRM. De Gruyter, 208-213. [Link]
Oostrom, J. K., Born, M. P., Serlie, A. W., & van der Molen, H. T. (2012). Implicit trait policies in multimedia situational judgment tests for leadership skills: Can they predict leadership behavior?. Human Performance, 25(4), 335-353. [Link]
Philpot, A., Devitt, R., Parr, J., & Nixon, B. (2002). Leadership competency models: roadmaps to success. Hospital Quarterly, 6(1), 42-45. [Link]
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and assessment, 11(1), 1-16. [Link]
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and assessment, 11(1), 1-16. [Link]
Ployhart, R. E., & MacKenzie Jr, W. I. (2011). Situational judgment tests: A critical review and agenda for the future. [Link]
Rafique, A., & Ghazal, S. (2022). Content Validation of a Situational Judgment Test of Manager’s Integrity: A Qualitative Approach. Journal of Professional & Applied Psychology, 3(2), 306-318. [Link]
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2021). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology. [Link]
Sander, N., & Muck, P. (2014). Answering the call for construct orientation: Developing a situational judgment test of achievement motivation. Personality and Individual Differences, 60, S26. [Link]
Saxena, A., Desanghere, L., Dore, K., & Reiter, H. (2024). Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes. BMC Medical Education, 24(1), 339. [Link]
Smith, K. J., Neely, S., Dennis, V. C., Miller, M. M., & Medina, M. S. (2022). Use of situational judgment tests to teach empathy, assertiveness, communication, and ethics. American Journal of Pharmaceutical Education, 86(6). [Link]
Steffensen Jr, D. S. (2014). Validation of a situational judgment test measuring teamwork processes (Doctoral dissertation, Middle Tennessee State University).rafiq [Link]
Stemler, S. E., & Sternberg, R. J. (2006). Using situational judgment tests to measure practical intelligence. Situational judgment tests: Theory, measurement, and application, 107-131. [Link]
Suen, H. Y., Chen, M. Y. C., & Lu, S. H. (2019). Does the use of synchrony and artificial intelligence in video interviews affect interview ratings and applicant attitudes?. Computers in Human Behavior, 98, 93-101. [Link]
Teng, Y., Brannick, M. T., & Borman, W. C. (2020). Capturing resilience in context: Development and validation of a situational judgment test of resilience. Human performance, 33(2-3), 74-103. [Link]
Thornton III, G. C., & Byham, W. C. (2013). Assessment centers and managerial performance. Elsevier. [Link]
Thornton III, G. C., Mueller-Hanson, R. A., & Rupp, D. E. (2017). Developing organizational simulations: A guide for practitioners, students, and researchers. Taylor & Francis. [Link]
Thornton III, G. C., Rupp, D. E., & Mueller-Hanson, R. A. (2003). Developing organizational simulations: A guide for practitioners and students. Psychology Press. [Link]
Tramontano, C., Grant, C., & Clarke, C. (2021). Development and validation of the e-Work Self-Efficacy Scale to assess digital competencies in remote working. Computers in human behavior reports, 4, 100129. [Link]
Wainer, H., & Braun, H. I. (Eds.). (2013). Test validity. Routledge. [Link]
Wasiluk, A. (2023). Assessment of managerial competencies by Generation Z. Zeszyty Naukowe Politechniki Śląskiej. Organizacja i Zarządzanie, (181). [Link]
Webster, E. S., Paton, L. W., Crampton, P. E., & Tiffin, P. A. (2020). Situational judgement test validity for selection: A systematic review and meta‐analysis. Medical Education, 54(10), 888-902. [Link]
Weekley, J. A., & Ployhart, R. E. (2013). Situational judgment tests: Theory, measurement, and application. Psychology Press. [Link]
Weekley, J. A., Hawkes, B., Guenole, N., & Ployhart, R. E. (2015). Low-fidelity simulations. Annu. Rev. Organ. Psychol. Organ. Behav. 2(1), 295-322. [Link]
Whetzel, D., Sullivan, T., & McCloy, R. (2020). Situational judgment tests: An overview of development practices and psychometric characteristics. Personnel Assessment and Decisions, 6 (1), Article Article 1. [Link]
Wong, S. C. (2020). Competency definitions, development and assessment: A brief review. International Journal of Academic Research in Progressive Education and Development, 9(3), 95-114. [Link]