Investigating The Psychometric Properties of The Situational Judgment Test for Assessing Employees' Job Competency

Document Type : Original Article

Authors

1 Ph.D. Student, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.

2 Assistant Prof, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.

3 Associate Prof, Faculty of Assessment and Measurement Allameh Tabataba'i University, Tehran, Iran.

4 Associate Prof, Faculty of Counseling, Allameh Tabataba'i University, Tehran, Iran.

Abstract

Aim: This study aimed to assess the psychometric properties of a situational judgment test (SJT) designed to evaluate job-related competencies among employees.

Method: This applied, instrument-development study was conducted with 300 private-sector employees in Iran, selected via convenience sampling. A researcher-developed SJT assessed competencies like responsibility and communication. Analyses included Cronbach’s alpha for reliability, item difficulty/discrimination indices, and confirmatory factor analysis using SPSS-23, LISREL 8.8, and Jmetrik V4.0.5.

Results: The SJT demonstrated good reliability and validity. Normative scores were developed for the target population.

Conclusion: The SJT is a reliable and valid tool for identifying behavioral competencies and is recommended for primary and secondary screening in employee selection.

Keywords

Main Subjects

References

Allen, J. G. (2004). The complete Q&A job interview book. John Wiley & Sons. [Link]
Bergman, M. E., Drasgow, F., Donovan, M. A., Henning, J. B., & Juraska, S. E. (2006). Scoring situational judgment tests: Once you get the data, your troubles begin. International Journal of Selection and Assessment, 14(3), 223-235. [Link]
Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative and its relationship to performance. Personnel Psychology, 62(2), 229-258. [Link]
Borchert, R. J., Hickman, C. R., Pepys, J., & Sadler, T. J. (2023). Performance of ChatGPT on the situational judgement test—a professional dilemmas–based examination for doctors in the United Kingdom. JMIR Medical Education, 9(1), e48978. [Link]
Brady, M. (2019). A Situational Judgment Test of Self-Control and its Relationship to Academic Performance: Development of a New Measure. [Link]
Brooks, M. E., Highhouse, S., Weekley, J. A., & Ployhart, R. E. (2006). Can good judgment be measured? Situational judgment tests: Theory, measurement, and application, 39-55. [Link]
Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta‐analysis of their criterion‐related validities. Personnel Psychology, 63(1), 83-117. [Link]
Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Harvey, V. S. (2001). Incremental validity of situational judgment tests. Journal of Applied Psychology, 86(3), 410. [Link]
Drucker, P. (2012). Managing in a time of great change. Routledge. [Link]
Flick, U. (2021). Doing interview research: The essential how to guide. [Link]
Freudenstein, J. P., Schäpers, P., Reznik, N., Stolte, T., & Krumm, S. (2024). The influence of situational strength on the relation of personality and situational judgment test performance. International Journal of Selection and Assessment, 32(1), 1-11. [Link]
Golubovich, J., Lake, C. J., Anguiano-Carrasco, C., & Seybert, J. (2020). Measuring achievement striving via a situational judgment test: The value of additional context. Revista de Psicología Del Trabajo y de las Organizaciones, 36(2), 157-168. [Link]
Golubovich, J., Seybert, J., Martin-Raugh, M., Naemi, B., Vega, R. P., & Roberts, R. D. (2017). Assessing perceptions of interpersonal behavior with a video-based situational judgment test. International Journal of Testing, 17(3), 191-209. [Link]
Grossman, G., & Sharf, R. (2018). Situational Judgment Tests and Transformational Leadership: An Examination of the Decisions, Leadership, and Experience in Undergraduate Leadership Development. Journal of Leadership Education, 17(1). [Link]
Harwood, H., Roulin, N., & Iqbal, M. Z. (2024). “Anything you can do, I can do”: Examining the use of ChatGPT in situational judgement tests for professional program admission. Journal of Vocational Behavior, 154, 104013. [Link]
Haynes, S. N., & Lench, H. C. (2003). Incremental validity of new clinical assessment measures. Psychological assessment, 15(4), 456. [Link]
Hedlund, J., Forsythe, G. B., Horvath, J. A., Williams, W. M., Snook, S., & Sternberg, R. J. (2003). Identifying and assessing tacit knowledge: Understanding the practical intelligence of military leaders. The leadership quarterly, 14(2), 117-140. [Link]
Heininger, S. K., Baumgartner, M., Zehner, F., Burgkart, R., Söllner, N., Berberat, P. O., & Gartmeier, M. (2021). Measuring hygiene competence: the picture-based situational judgement test HygiKo. BMC medical education, 21(1), 410. [Link]
Hernandez, J., Muratet, M., Pierotti, M., & Carron, T. (2022, July). Enhancement of a Gamified Situational Judgment Test Scoring System for Behavioral Assessment. In 2022 International Conference on Advanced Learning Technologies (ICALT) (pp. 374-378). IEEE. [Link]
Hickman, L., Bosch, N., Ng, V., Saef, R., Tay, L., & Woo, S. E. (2022). Automated video interview personality assessments: Reliability, validity, and generalizability investigations. Journal of Applied Psychology, 107(8), 1323. [Link]
Hoffman, B. J., Kennedy, C. L., LoPilato, A. C., Monahan, E. L., & Lance, C. E. (2015). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology, 100(4), 1143. [Link]
Jesiek, B. K., Woo, S. E., Parrigon, S., & Porter, C. M. (2020). Development of a situational judgment test for global engineering competency. Journal of Engineering Education, 109(3), 470-490. [Link]
Khosrojedi, Z., & Heidari, M. (2024). Research Article The Psychometric Characteristics of Strength‐Based Parenting Scale, Parent Child Version. Applied Psychology, 18(2), 86-106. [Link]
Kumazaki, H., Yoshikawa, Y., Muramatsu, T., Haraguchi, H., Fujisato, H., Sakai, K., ... & Mimura, M. (2021). Group-Based Online Job Interview Training Program Using Virtual Robot for Individuals With Autism Spectrum Disorders. Frontiers in psychiatry, 12. [Link]
Landers, R. N., Auer, E. M., & Abraham, J. D. (2020). Gamifying a situational judgment test with immersion and control game elements: Effects on applicant reactions and construct validity. Journal of Managerial Psychology, 35(4), 225-239. [Link]
Langer, M., König, C. J., & Papathanasiou, M. (2019). Highly automated job interviews: Acceptance under the influence of stakes. International Journal of Selection and Assessment, 27(3), 217-234. [Link]
Lievens, F., & Sackett, P. R. (2006). Video-based versus written situational judgment tests: a comparison in terms of predictive validity. Journal of applied psychology, 91(5), 1181. [Link]
Lievens, F., & Sackett, P. R. (2012). The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. Journal of Applied Psychology, 97(2), 460. [Link]
Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426-441. [Link]
McDaniel, M. A., Hartman, N. S., Whetzel, D. L., & GRUBB III, W. L. (2007). Situational judgment tests, response instructions, and validity: A meta‐analysis. Personnel psychology, 60(1), 63-91. [Link]
McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., Campion, M. A., & Braverman, E. P. (2001). Use of situational judgment tests to predict job performance: a clarification of the literature. Journal of Applied Psychology, 86(4), 730. [Link]
Meyer, R. D., Dalal, R. S., & Hermida, R. (2010). A review and synthesis of situational strength in the organizational sciences. Journal of management, 36(1), 121-140. [Link]
Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (2006). An alternative selection procedure: The low-fidelity simulation. Journal of applied psychology, 75(6), 640. [Link]
Murano, D., Lipnevich, A. A., Walton, K. E., Burrus, J., Way, J. D., & Anguiano-Carrasco, C. (2021). Measuring social and emotional skills in elementary students: Development of self-report Likert, situational judgment test, and forced choice items. Personality and Individual Differences, 169, 110012. [Link]
Nørskov, S., & Ulhøi, J. P. (2020). The use of robots in job interviews. Encyclopedia of electronic HRM. De Gruyter, 208-213. [Link]
Oostrom, J. K., Born, M. P., Serlie, A. W., & van der Molen, H. T. (2012). Implicit trait policies in multimedia situational judgment tests for leadership skills: Can they predict leadership behavior?. Human Performance, 25(4), 335-353. [Link]
Philpot, A., Devitt, R., Parr, J., & Nixon, B. (2002). Leadership competency models: roadmaps to success. Hospital Quarterly, 6(1), 42-45. [Link]
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and assessment, 11(1), 1-16. [Link]
Ployhart, R. E., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and assessment, 11(1), 1-16. [Link]
Ployhart, R. E., & MacKenzie Jr, W. I. (2011). Situational judgment tests: A critical review and agenda for the future. [Link]
Rafique, A., & Ghazal, S. (2022). Content Validation of a Situational Judgment Test of Manager’s Integrity: A Qualitative Approach. Journal of Professional & Applied Psychology, 3(2), 306-318. [Link]
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2021). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology. [Link]
Sander, N., & Muck, P. (2014). Answering the call for construct orientation: Developing a situational judgment test of achievement motivation. Personality and Individual Differences, 60, S26. [Link]
Saxena, A., Desanghere, L., Dore, K., & Reiter, H. (2024). Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes. BMC Medical Education, 24(1), 339. [Link]
Smith, K. J., Neely, S., Dennis, V. C., Miller, M. M., & Medina, M. S. (2022). Use of situational judgment tests to teach empathy, assertiveness, communication, and ethics. American Journal of Pharmaceutical Education, 86(6). [Link]
Steffensen Jr, D. S. (2014). Validation of a situational judgment test measuring teamwork processes (Doctoral dissertation, Middle Tennessee State University).rafiq [Link]
Stemler, S. E., & Sternberg, R. J. (2006). Using situational judgment tests to measure practical intelligence. Situational judgment tests: Theory, measurement, and application, 107-131. [Link]
Suen, H. Y., Chen, M. Y. C., & Lu, S. H. (2019). Does the use of synchrony and artificial intelligence in video interviews affect interview ratings and applicant attitudes?. Computers in Human Behavior, 98, 93-101. [Link]
Teng, Y., Brannick, M. T., & Borman, W. C. (2020). Capturing resilience in context: Development and validation of a situational judgment test of resilience. Human performance, 33(2-3), 74-103. [Link]
Thornton III, G. C., & Byham, W. C. (2013). Assessment centers and managerial performance. Elsevier. [Link]
Thornton III, G. C., Mueller-Hanson, R. A., & Rupp, D. E. (2017). Developing organizational simulations: A guide for practitioners, students, and researchers. Taylor & Francis. [Link]
Thornton III, G. C., Rupp, D. E., & Mueller-Hanson, R. A. (2003). Developing organizational simulations: A guide for practitioners and students. Psychology Press. [Link]
Tramontano, C., Grant, C., & Clarke, C. (2021). Development and validation of the e-Work Self-Efficacy Scale to assess digital competencies in remote working. Computers in human behavior reports, 4, 100129. [Link]
Wainer, H., & Braun, H. I. (Eds.). (2013). Test validity. Routledge. [Link]
Wasiluk, A. (2023). Assessment of managerial competencies by Generation Z. Zeszyty Naukowe Politechniki Śląskiej. Organizacja i Zarządzanie, (181). [Link]
Webster, E. S., Paton, L. W., Crampton, P. E., & Tiffin, P. A. (2020). Situational judgement test validity for selection: A systematic review and meta‐analysis. Medical Education, 54(10), 888-902. [Link]
Weekley, J. A., & Ployhart, R. E. (2013). Situational judgment tests: Theory, measurement, and application. Psychology Press. [Link]
Weekley, J. A., Hawkes, B., Guenole, N., & Ployhart, R. E. (2015). Low-fidelity simulations. Annu. Rev. Organ. Psychol. Organ. Behav. 2(1), 295-322. [Link]
Whetzel, D., Sullivan, T., & McCloy, R. (2020). Situational judgment tests: An overview of development practices and psychometric characteristics. Personnel Assessment and Decisions, 6 (1), Article Article 1. [Link]
Wong, S. C. (2020). Competency definitions, development and assessment: A brief review. International Journal of Academic Research in Progressive Education and Development, 9(3), 95-114. [Link]

Files

Share

How to cite

Statistics