The Residency Certification Exams – The Department Effect on the Otolaryngology Head and Neck Surgery Theoretical and Clinical Exams: A National Survey and Policy Implications

Nir Hirshoren, MD, MHA¹˒²˒³; Hila Porat, MSc²; Tali Landau Zemer, MD¹˒²; Michal Shauly-Aharonov, PhD⁴˒⁵; Ron Eliashar, MD¹˒²

1. Department of Otolaryngology / Head & Neck Surgery, Hadassah Hebrew-University Medical Center, Jerusalem, Israel
2. Faculty of Medicine, Hebrew University of Jerusalem, Jerusalem, Israel
3. Institute for Research of Military Medicine, Faculty of Medicine, Hebrew University of Jerusalem, Jerusalem, Israel
4. School of Public Health and Community Medicine, Hebrew University of Jerusalem, Jerusalem, Israel
5.  Department of Industrial Engineering and Management, Jerusalem College of Technology, Jerusalem, Israel

OPEN ACCESS

PUBLISHED: 31 October 2024

CITATION: Hirshoren, N., Porat, H., et al., 2024. The Residency Certification Exams – The Department Effect on the Otolaryngology Head and Neck Surgery Theoretical and Clinical Exams: A National Survey and Policy Implications. Medical Research Archives, [online] 12(10).https://doi.org/10.18103/mra.v12i10.5828

COPYRIGHT: © 2024 European Society of Medicine. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

DOI https://doi.org/10.18103/mra.v12i10.5828

ISSN 2375-1924

Abstract

Background: Otorhinolaryngology / Head and Neck Surgery residents in Israel face two certification exams during residency, the first (Step A), a theoretical exam, and the second (Step B), a clinical exam. The exams require different qualifications in order to succeed in each. Herein, we investigate the institutional “effect” on the theoretical and clinical certification exams.

Aims: The study Objectives are: 1. To investigate the correlation between institutional workload and surgical training (“effect”) and the perception of being prepared to the theoretical and clinical certification exams. 2. To analyze residents’ comprehensive perception of competence according to their certification exams status.

Methods: An anonymous electronic questionnaire was emailed to all registered Israeli Otorhinolaryngology / Head and Neck Surgery residents. The questionnaire investigated the residents’ perception of their preparation for the certification exams with regard to their department’s workload, training, and competence in the five main sub-specialties: Otology, Head and Neck Surgery, Pediatric Ear-Nose and Throat, Rhinology and Laryngology.

Results: Overall, 92 (63.5%) residents responded to the questionnaire. Of these, 17 residents had successfully passed the theoretical, Step A, certification exam, and 14 passed the clinical, Step B, certification exam. We have demonstrated an institutional “effect” (workload and surgical training) in most sub-specialties in clinical Step B, which was less observed in theoretical Step A. In addition, Otorhinolaryngology / Head and Neck Surgery residents’ comprehensive perception (mainly clinical and surgical competence) was significantly higher following clinical Step B compared to their perception after theoretical Step A. These significant differences were found in all sub-specialties except Rhinology.

Conclusions: The clinical, Step B exam, is a highly important milestone in the residency program. The perception of being well prepared to clinical Step B is more affected by the workload and surgical training, varying between different institutions. We recommend having the clinical, Step B exam, earlier in the course of the residency program and suggesting the need for central training system and supervision (e.g. regional curriculum collaboration) for a better medical training.

Keywords: Training program; Residency; Otorhinolaryngology Head and Neck Surgery; Curriculum; Qualification exams

List of abbreviations:

• ORL-HNS; Otorhinolaryngology / Head and Neck Surgery
• ENT; Ear-Nose and Throat
• PGY; Postgraduate year
• USMLE; United States Medical Licensing Examination

Introduction:

Most residency programs in Israel lasts four to six years. Every resident faces two types of certification residency exams during this period. The first exam, Step A, performed after completing at least two years of the residency program, is a written theoretical exam, and is based on the professional literature. The aim of Step A exam is to enhance and to assess the theoretical knowledge of the resident. The second exam, Step B, is performed during the last year of residency. It may be taken only after successfully passing the theoretical, Step A, exam. Step B, clinical exam, evaluates the resident’s decision-making process and competence. It is a clinical exam, based on the resident’s ability to confront and solve clinical problems, analyze and integrate data, recommend the right treatment, and be able to carry out this treatment.

The two exams evaluate different qualifications. Theoretical exams are often time-consuming, requiring a high level of self-discipline, which correlates with higher scores. It is believed that most factors impacting the performance in a theoretical exam are related to the individual resident’s motivation and work ethic and less to the teaching institution. Clinical exams, on the other hand, require a different kind of qualification, relying on training and preparation. Adequate preparation results in significant improvement in confidence and competence. Diverse training programs, with variable institutional characteristics, may impact the performance and preparedness of their graduates for the certification exams.

Residency in Otorhinolaryngology / Head and Neck Surgery (ORL-HNS) in Israel lasts six years. There are five main ORL-HNS sub-specialties: Otology, Head and Neck Surgery, Pediatric Ear-Nose and Throat (ENT), Rhinology and Laryngology. The ORL-HNS training programs inherently vary between institutions, bearing in mind different workload and senior staff professionalism, differences between local and referral hospitals, and differences between peripheral and central medical centers. Herein, we investigate the institutional “effect” on the theoretical and clinical certification residency exams in ORL-HNS in Israel. In addition, we explore the confidence and professional maturity achieved following the qualification exams.

Methods:

The study was approved and conducted according to the local Internal Review Board guidelines (Helsinki Committee. 0621-20-HMO). An anonymous electronic questionnaire was emailed to all registered Israeli ORL-HNS residents. The questionnaire was assembled and tailored by professional medicine educators and public health and community medicine experts. Participants’ names and affiliations were not requested in order to ensure truthful answers. The questionnaire investigated the residents’ perception on their preparedness to the residency certification exams with regard to their department’s workload, training, and competence (in the five main ORL-HNS sub-specialties). We divided the training and competence perceptions into ‘surgical’ and ‘clinical’. While ‘surgical’ stands for technical considerations and operating skills, ‘clinical’ means knowledge necessary for diagnosis, investigation (e.g., laboratory, imaging, and pathology), treatment, and follow-up. In each question, the participant was asked to grade his / her perception from 1 to 10, where grades 1-2 stood for “very low”, 3-4 for “low”, 5-6 for “average”, 7-8 for “high”, and 9-10 for “very high”.

PARTICIPANTS:

We approached anonymously all registered ORL-HNS residents, postgraduate year, PGY, 1 to 6. We categorized the residents into three groups according to their certification exams status: Before performing the theoretical Step A; after successfully achieving the theoretical Step A but before performing the clinical Step B; and after successful achievements of the clinical Step B.

PRIMARY GOALS:

1. To investigate the correlation between workload and surgical training (department “effect”) and the perception of being well prepared for the theoretical and clinical certification exams.
2. To analyze residents’ comprehensive readiness perception (training and competence) in different stages during the residency program, according to their achievements in the residency certification exams.

STATISTICAL ANALYSIS:

Statistical analysis was performed using R software, version 3.5.1 (R Development Core Team, 2018) and Microsoft Excel. Basic participants’ perceptions were summarized using means, medians, and inter-quartile ranges when appropriate. Pearson correlations between the different characteristics were calculated and their p-values were corrected for multiple comparisons using Holm’s correction. Wilcoxon-Mann-Whitney test was performed to compare the perceptions of residents after the theoretical Step A exam, to those of residents after the clinical Step B exam. P-value of 0.05 or less was considered statistically significant.

Results:

Overall, 92 residents out of 145 registered ORL-HNS residents in Israel (63.5%) responded to the electronic questionnaire. Of these, 17 residents had successfully passed the theoretical, Step A, certification exam, and 14 passed the clinical, Step B, certification exam.

All correlations between the perception of being well prepared for the exams and the department characteristics (workload and surgical training) were higher (in all sub-specialties) in the clinical Step B compared to the theoretical Step A exam.

Asterisk(*) stands for a significant correlation. H & N Surgery: Head and Neck Surgery; Pediatric ENT: Pediatric Ear-Nose and Throat

The corresponding p-values reached significance in the theoretical Step A exam only when correlating the resident’s preparedness to the exam with workload in laryngology (p=0.012) and surgical training in head and neck surgery and laryngology (p=0.007 and p=0.001, respectively). Whereas, statistical significance was achieved in the clinical Step B exam in more subspecialties; correlating workload in otology and laryngology (p=0.005 and p=0.048, respectively) and correlating surgical training in otology, head and neck surgery and laryngology (p=0.001, p=0.005 and p=0.001, respectively).

The ORL-HNS residents’ comprehensive readiness perception as emerged by investigating the clinical and surgical training, and clinical and surgical competence, was significantly higher following the clinical, Step B, exam, compared to the theoretical, Step A, exam. These significant differences were implicated in all sub-specialties except in Rhinology.

Discussion:

Examinations are an integral and essential part of the process of medical specialization. Residents cannot receive a Specialist degree without passing formal exams. The residency exams in Israel are performed in two phases: theoretical Step A (written examination) followed by clinical Step B (oral examination).

In the current study we have demonstrated a more institutional “effect” in Step B, the clinical certification exam, in most sub-specialties, which was less apparent (lower correlation) in the theoretical, Step A, exam. Moreover, we have clearly shown a “maturity” effect following a successful achievement in clinical, Step B. Various factors modify the residents’ perception for being well prepared for the certification exams. Obviously, the individual’s effort, persistence and personal abilities, highly influence the performance in these exams. However, we cannot ignore the potential difference relating to the institution in which the resident is trained. We have termed the department work load and training (clinical and surgical), which vary in different institutes, as department “effect”. This is a qualitative and quantitative definition. Herein we highlight measurable department characteristics such as volume (work load), training, faculty (residents) satisfaction. The residency programs ranking is a tool expressing departments’ “effect”. The Doximity residency match survey is an example of a comprehensive ranking tool based on physician only network. Interesting investigations have demonstrated that the residents’ performance in the American Board of Surgery Certifying Examination (ABSCE) correlated with training programs consisting simulations with senior staff.

Few health systems overcome these institutional variations by joining hospitals into combined training systems, in which the residents work and learn by rotating in all departments.

Separate medical theoretical and clinical exams are common, both during and prior to residency. United States Medical Licensing Examination, USMLE Step 1 consists of multiple-choice questions assessing the examinee’s basic science knowledge. Similar to the Step A exam, USMLE Step 1 is a theoretical exam. Individual efforts, including a long study period, increased book usage and practice of multiple-choice questions, are all associated with higher USMLE Step 1 scores. Previously published data displayed an association between USMLE Step 1 scores and successful first-time passage of the American Board of Otolaryngology Written Qualifying Examination. Both are theoretical exams, more dependent on individual character traits. In the same manner, our current study demonstrates low correlation between being well prepared for the theoretical Step A exam and department characteristics (workload and surgical training) in almost all sub-specialties. This finding supports the action recently taken by the National Higher Examination Committee and the Scientific Council, which has enabled all residents, no matter how long the residency period is, to perform the theoretical Step A exams earlier along the residency period.

In contrast to the theoretical Step 1, USMLE Step 2 is divided into two parts: clinical knowledge and clinical skills. Both parts assess the ability of the examinees to apply medical knowledge, medical skills, and understanding of clinical science. Likewise, Step B is a clinical exam. Clinical exams are more likely to be affected by the residents’ training program and ongoing exposure, as clinical skills are not achieved merely by reading. USMLE Step 2 exam was not as strong a predictor of success in the written exam. On the other hand, incorporating structured clinical mock oral examinations during the residency program correlated with successful first-time passage rates of certifying clinical examinations. Similarly, we identified that a higher perception of being well prepared to the clinical residency Step B exam correlated with the department the resident was coming from, as characterized by workload and surgical training.

Thereafter, we have investigated the “maturity”, readiness, the residents gain following the different exams. A higher comprehensive perception, mainly clinical and surgical competence, was noted following the clinical exam. Therefore, Step B seems to be a turning point in the training program. Interestingly, clinical and surgical training, together with competence perception, was high in rhinology following both theoretical and clinical exams. This finding could be related to a relatively fast learning curve in rhinology, as result of a better hands-on and a better-visualized and supervised subspecialty. This actually highlight the importance of curriculum planning and precision, considering the requirements in the different sub-specialties. A complementary advantage to having a better residency program is a reduction in burnout feelings.

Our investigation main limitation is related to the small study groups considering that the certification exams are currently been held during the last years of the program. Therefore, 61 participants could not shed light on the study goals. This is inherited situation in relative small health system, which we believe we overcome by high responsive rate.

Conclusions

The residency clinical exam, is a highly important milestone in the residency program. The perception of being well prepared to the clinical, Step B, exam, is exceedingly affected by the workload and surgical training, varying between different institutions. We encourage each surgical and non-surgical health profession to analyze the institutional “effect”, especially on the clinical exam, which may force curriculum adjustments/ modifications, or even persuade a more radical structural revolution forming a central training system, not institutionally dependent.

Declarations:

ETHICS APPROVAL AND CONSENT TO PARTICIPATE:

The study was conducted following the approval by the institutional review board (0621-20- HMO), Hadassah medical center. The study was conducted according to the 2008 World Medical Association Declaration of Helsinki. Participants had to approve their participation (this was a digital informed consent obtained from all participants). The participants were aware of the study’s purpose.

CONSENT FOR PUBLICATION:

Not applicable. The information / data is not identifiable.

AVAILABILITY OF DATA AND MATERIALS:

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

COMPETING INTERESTS:

All authors declare that they have no competing interests.

FUNDING:

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHORS’ CONTRIBUTIONS:

N.H: conceptualization, analysis, methodology, project administration, supervision, validation, writing, agree for publication. H.P: conceptualization, data curation, methodology, reviewing, discussion and agree for publication. T.L.Z: conceptualization, analysis, methodology, project administration, supervision, validation, writing, agree for publication. M.S.A: conceptualization, project administration, supervision, validation, writing, agree for publication. R.E: conceptualization, methodology, supervision, validation, reviewing, agree for publication.

ACKNOWLEDGMENT:

Not applicable.

References:

1. Kumar AD, Shah MK, Maley JH, Evron J, Gyftopoulos A, Miller C. Preparing to take the USMLE Step 1: A survey on medical students’ self-reported study habits. Postgrad Med J. 2015;91(1075):257-261. Doi:10.1136/postgradmedj-2014-133081
2. Burk-Rafel J, Santen SA, Purkiss J. Study Behaviors and USMLE Step 1 Performance: Implications of a Student Self-Directed Parallel Curriculum. Acad Med. 2017;92(11):S67-S74. Doi: 10.1097/ACM.0000000000001916
3. Levine RB, Levy AP, Lubin R, Halevi S, Rios R, Cayea D. Evaluation of a course to prepare international students for the United States Medical Licensing Examination step 2 clinical skills exam. J Educ Eval Health Prof. 2017;14:25. Doi: 10.3352/jeehp.2017.14.25
4. Touma NJ, Beiko DT, MacNeily AE, Leveridge MJ. Impact of a training program on the performance of graduating Canadian residents on a national urology exam: Results of the last 20 years. Can Urol Assoc J. 2019;13(2):39-42. Doi:10.5489/cuaj.5386
5. Oker N, Alotaibi NH, Reichelt AC, Herman P, Bernal-Sprekelsen M, Albers AE. European otorhinolaryngology training programs: results of a European survey about training satisfaction, work environment and conditions in six countries. Eur Arch Oto-Rhino-Laryngology. 2017;274(11):4017-4029. Doi: 10.1007/s00405-017-4727-0
6. Oker N, Escabasse V, Pensky H, Alotaibi N, Coste A, Albers AE. Training satisfaction and work environment in Otorhinolaryngology, Head and Neck surgery: A comparison between France and Germany. Eur Arch Oto-Rhino-Laryngology. 2014;271(9):2565-2573. Doi: 10.1007/s00405-014-3046-y
7. Burke LG, Frakt AB, Khullar D, Orav EJ, Jha AK. Association between teaching status and mortality in US Hospitals. JAMA – J Am Med Assoc. 2017;317(20):2105-2113. Doi: 10.1001/jama.2017.5702
8. Eliashar R. Residency examinations – The activity of the Higher Examination Committee of the Scientific Council for the training of the next generation Israeli physicians. Harefuah. 2019;158(10):639-642.
9. Guzman E, Babakhani A, Maker VK. Improving Outcomes on the ABS Certifying Examination: Can Monthly Mock Orals Do It? J Surg Educ. 2008;65(6):441-444. Doi:10.1016/j.jsurg.2008.04.005
10. Fingeret AL, Arnell T, McNelis J, Statter M, Dresner L, Widmann W. Sequential Participation in a Multi-Institutional Mock Oral Examination Is Associated With Improved American Board of Surgery Certifying Examination First-Time Pass Rate. J Surg Educ. 2016;73(6):e95-e103. Doi:10.1016/j.jsurg.2016.06.016
11. Villwock JA, Hamill CS, Ryan JT, Nicholas BD. The Role of the Away Rotation in Otolaryngology Residency. Otolaryngol – Head Neck Surg (United States). 2017;156(6):1104-1107. Doi:10.1177/0194599817698431
12. Bruening MH, Anthony AA, Maddern GJ. Surgical rotations in provincial South Australia: The trainees’ perspective. ANZ J Surg. 2003;73(1-2):65-68. Doi:10.1046/j.1445-2197.2003.02620.x
13. Perez JA, Greer S. Correlation of United States medical licensing examination and internal medicine in-training examination performance. Adv Heal Sci Educ. 2009;14(5):753-758. Doi:10.1007/s10459-009-9158-2
14. Puscas L, Chang CWD, Lee HJ, Diaz R, Miller R. USMLE and Otolaryngology: Predicting Board Performance. Otolaryngol – Head Neck Surg (United States). 2017;156(6):1130-1135. Doi:10.1177/0194599817702874
15. Geelan-Hansen K, Anne S, Benninger MS. Burnout in Otolaryngology-Head and Neck Surgery: A Single Academic Center Experience. Otolaryngol Head Neck Surg. 2018 Aug;159(2):254-257. Doi: 10.1177/0194599818774740. PMID: 29759029.
16. Kim DI, Loo LK, Garrison RC, et al. Does teaching Optimism lower Burnout in residency training- a pilot study. J Community Hosp Intern Med Perspect. 2021 Jun 21;11(4):429-432. Doi: 10.1080/20009666.2021.1910408. PMID: 34211643.