Thu, Feb 20, 2025
[Archive]
Volume 17, Issue 56 (2024)
JMED 2024, 17(56): 87-96 |
Back to browse issues page
Ethics code: (UOC/CRERC/ EXEMPTION/01/2023)
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Ansari R, Ab Manan N, Mahat N A, Omar N S, Abdul Latiff A, Idris S et al . Standard setting methods in objective structured clinical examination (OSCE): A comparative study of five methods. JMED 2024; 17 (56) :87-96
URL: http://edujournal.zums.ac.ir/article-1-2116-en.html
URL: http://edujournal.zums.ac.ir/article-1-2116-en.html
Reshma Ansari *1 
, Norhafizah Ab Manan2 , Nur Ain Mahat3 , Norfaizatul Shalida Omar4 , Atikah Abdul Latiff5 , Sara Idris5 , Azli Shahril Othman6
, Norhafizah Ab Manan2 , Nur Ain Mahat3 , Norfaizatul Shalida Omar4 , Atikah Abdul Latiff5 , Sara Idris5 , Azli Shahril Othman6
1- Medical Education Department International Medical School Management & Science University, Selangor Malaysia , reshmaansari77@gmail.com
2- Department of Public Health Faculty of Medicine University of Cyberjaya, Selangor Malaysia
3- Kulliyah of Nursing International Islamic University, Kuantan, Pahang, Malaysia.
4- Department of Biochemistry Manipal University College Malaysia, Malacca Malaysia
5- Department of Anatomy Faculty of Medicine University of Cyberjaya, Selangor Malaysia
6- Department of Pharmacology Faculty of Medicine and Defence Health, National Defence University Malaysia, Kuala Lumpur Malaysia
2- Department of Public Health Faculty of Medicine University of Cyberjaya, Selangor Malaysia
3- Kulliyah of Nursing International Islamic University, Kuantan, Pahang, Malaysia.
4- Department of Biochemistry Manipal University College Malaysia, Malacca Malaysia
5- Department of Anatomy Faculty of Medicine University of Cyberjaya, Selangor Malaysia
6- Department of Pharmacology Faculty of Medicine and Defence Health, National Defence University Malaysia, Kuala Lumpur Malaysia
Abstract: (300 Views)
Background & Objective: Objective Structured Clinical Examination (OSCE) is a crucial component in medical school examinations to assess students’ competency, particularly in clinical skills incorporating cognitive and affective domains. OSCE results are subjected to standard-setting methods, which yield different findings. Hence, in this study, five different standard-setting methods, namely norm reference, Angoff method, borderline group method (BGM), borderline regression method (BRM), and modified Cohen’s method, were compared to determine the cut-off scores and failure rates determined by each method.
Material & Methods: Data of 170 second-year medical students who attended OSCE with eight stations for their First Professional Examination at the end of year 2 MBBS was taken for the study following ethical approval. Total scores for each station were standardized to 20 marks, and cut-off scores were determined using each of the five standard-setting methods.
Results: As a comparison of 5 methods, the Norm reference method yielded the highest number of stations with high cut-off scores, followed by BRM. This is reflected in the number of failures, too. On the contrary, using the Angoff method yielded the lowest cut-off scores in maximum stations, resulting in the least number of failed students. The Cochrane’s Q test of the results yielded a p < 0.001, which signifies that the proportion of students who failed a particular OSCE station was significantly different when different methods were used to determine the cut score.
Conclusion: The study, which compared 5 common standard-setting methods employed in medical education assessments, found that norm-referenced and BRM had high cut-off scores and failures, with the opposite determined by the Modified Angoff method. The study concluded that the cut-off score and failure rate differed with different standard-setting methods, and the choice of the method is contextual depending on the available resources.
Material & Methods: Data of 170 second-year medical students who attended OSCE with eight stations for their First Professional Examination at the end of year 2 MBBS was taken for the study following ethical approval. Total scores for each station were standardized to 20 marks, and cut-off scores were determined using each of the five standard-setting methods.
Results: As a comparison of 5 methods, the Norm reference method yielded the highest number of stations with high cut-off scores, followed by BRM. This is reflected in the number of failures, too. On the contrary, using the Angoff method yielded the lowest cut-off scores in maximum stations, resulting in the least number of failed students. The Cochrane’s Q test of the results yielded a p < 0.001, which signifies that the proportion of students who failed a particular OSCE station was significantly different when different methods were used to determine the cut score.
Conclusion: The study, which compared 5 common standard-setting methods employed in medical education assessments, found that norm-referenced and BRM had high cut-off scores and failures, with the opposite determined by the Modified Angoff method. The study concluded that the cut-off score and failure rate differed with different standard-setting methods, and the choice of the method is contextual depending on the available resources.
Keywords: standard setting, OSCE, Angoff, norm-reference, borderline group, borderline regression, Cohen
Article Type : Orginal Research |
Subject:
Medical Education
Received: 2024/01/31 | Accepted: 2024/08/20 | Published: 2024/12/14
Received: 2024/01/31 | Accepted: 2024/08/20 | Published: 2024/12/14
References
1. Ataro G. Methods, methodological challenges and lesson learned from phenomenological study about OSCE experience: overview of paradigm-driven qualitative approach in medical education. Annals of Medicine and Surgery. 2020;49:19-23. [DOI]
2. Khan KZ, Ramachandran S, Gaunt K, Pushkar P. The objective structured clinical examination (OSCE): AMEE guide no. 81. Part I: an historical and theoretical perspective. Medical Teacher. 2013;35(9):e1437-46. [DOI]
3. Hejri SM, Jalili M. Standard setting in medical education: fundamental concepts and emerging challenges. Medical Journal of the Islamic Republic of Iran. 2014;28:34.
4. Malau-Aduli BS, Teague PA, D’Souza K, et al. A collaborative comparison of objective structured clinical examination (OSCE) standard setting methods at Australian medical schools. Medical Teacher. 2017;39(12):1261-7. [DOI]
5. Elfaki OA, Salih KM. Comparison of two standard setting methods in a medical students MCQs exam in internal medicine. American Journal of Medicine and Medical Sciences. 2015;5(4):164-7. [DOI]
6. Abd‐Rahman AN, Baharuddin IH, Abu‐Hassan MI, Davies SJ. A comparison of different standard‐setting methods for professional qualifying dental examination. Journal of Dental Education. 2021;85(7):1210-6. [DOI]
7. Tavakol M, O'Brien D, Stewart C. Determining intra-standard-setter inconsistency in the Angoff method using the three-parameter item response theory. International Journal of Medical Education. 2023;14:123. [DOI]
8. Cohen-Schotanus J, van der Vleuten CP. A standard setting method with the best performing students as point of reference: practical and affordable. Medical Teacher. 2010;32(2):154-60. [DOI]
9. Saaiq M. Standard setting methods for the assessment of knowledge and skills in medical education. Journal of Health Professions Education and Innovation. 2024;1(2):14-20. [DOI]
10. Kamal D, Sallam M, Gouda E, Fouad S. “Is there a “best” method for standard setting in OSCE exams? Comparison between four methods (a cross-sectional descriptive study). Journal of Medical Education. 2020;19(1). [DOI]
11. Dwyer T, Wright S, Kulasegaram KM, et al. How to set the bar in competency-based medical education: standard setting after an objective structured clinical examination (OSCE). BMC Medical Education. 2016;16:1-7. [DOI]
12. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education. 2004;38(2):199-203. [DOI]
13. Smee S, Coetzee K, Bartman I, Roy M, Monteiro S. OSCE standard setting: three borderline group methods. Medical Science Educator. 2022;32(6):1439-45. [DOI]
14. Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, Van Der Vleuten C. Comparison of a rational and an empirical standard setting procedure for an OSCE. Medical Education. 2003;37(2):132-9. [DOI]
15. Elabd K, Abdul-Kadir H, Alkhenizan A, Alkhalifa MK. A Comparison of the checklist scoring systems, global rating systems, and borderline regression method for an objective structured clinical examination for a small cohort in a Saudi medical school. Cureus. 2023;15(6). [DOI]
16. Taylor CA. Development of a modified Cohen method of standard setting. Medical Teacher. 2011;33(12):e678-82. [DOI]
17. 17. McLachlan JC, Robertson KA, Weller B, Sawdon M. An inexpensive retrospective standard setting method based on item facilities. BMC Medical Education. 2021;21:1-7. [ [DOI]
18. 18. Khalid MN, Shafiq F, Ahmed S. A Comparison of standard setting methods for setting cut-scores for assessments with constructed response questions. Pakistan Journal of Educational Research and Evaluation. 2021;9(2):74-85.
19. 19. Goldenberg M, Ordon M, Honey JR, Andonian S, Lee JY. Objective assessment and standard setting for basic flexible ureterorenoscopy skills among urology trainees using simulation-based methods. Journal of Endourology. 2020;34(4):495-501. [DOI]
20. Martínez-Mesa J, González-Chica DA, Duquia RP, Bonamigo RR, Bastos JL. Sampling: how to select participants in my research study? Anais Brasileiros de Dermatologia. 2016;91(3):326-30. [DOI]
21. George S, Haque MS, Oyebode F. Standard setting: comparison of two methods. BMC Medical Education. 2006;6:1-6. [DOI]
22. Chong L, Taylor S, Haywood M, Adelstein BA, Shulruf B. The sights and insights of examiners in objective structured clinical examinations. Journal of Educational Evaluation for Health Professions. 2017;14. [DOI]
23. Dwivedi NR, Vijayashankar NP, Hansda M, et al. Comparing standard setting methods for objective structured clinical examinations in a Caribbean medical school. Journal of Medical Education and Curricular Development. 2020;7:2382120520981992. [DOI]
24. Park SY, Lee SH, Kim MJ, Ji KH, Ryu JH. Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea. Journal of Educational Evaluation for Health Professions. 2021;18;25. [DOI]
25. Wood TJ, Humphrey-Murto SM, Norman GR. Standard setting in a small scale OSCE: a comparison of the modified borderline-group method and the borderline regression method. Advances in Health Sciences Education. 2006;11:115-22. [DOI]
26. Wayne DB, Fudala MJ, Butter J, et al. Comparison of two standard-setting methods for advanced cardiac life support training. Academic Medicine. 2005;80(10):S63-6. [DOI]
27. Carlson J, Tomkowiak J, Knott P. Simulation-based examinations in physician assistant education: a comparison of two standard-setting methods. The Journal of Physician Assistant Education. 2010;21(2):7-14. [DOI]
28. Jalili M, Mortazhejri S. Standard setting for objective structured clinical exam using four methods: Prefixed score Angoff borderline regression and Cohens. Strides in Development of Medical Education. 2012;9(1):77-84.
29. Kamal D, ElAraby S, Kamel MH, Hosny S. Evaluation of two applied methods for standard setting in undergraduate medical programme at the faculty of medicine, Suez Canal university. Education in Medicine Journal. 2018;10(2):15–25. [DOI]
30. Yim MK, Shin S. Using the Angoff method to set a standard on mock exams for the Korean nursing licensing examination. Journal of Educational Evaluation for Health Professions. 2020;17(14):1149169. [DOI]
31. Kim DH, Kang YJ, Park HK. Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean medical licensing examination written test: a descriptive study. Journal of Educational Evaluation for Health Professions. 2022;19:33. [DOI]
32. Verheggen MM, Muijtjens AM, Van Os J, Schuwirth LW. Is an Angoff standard an indication of minimal competence of examinees or of judges? Advances in Health Sciences Education. 2008;13:203-11. [DOI]
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
