1.5.3. Performance Standard

Scoring for the Oral Examination is based on a candidate’s ability to appropriately diagnose and manage the patients described in the 42 clinical scenarios presented, all of which are equally weighted. Results are based on whether each candidate meets the overall passing standard. Scores are not adjusted in relation to how well other candidates perform, and though statistically unlikely, this method makes it possible for all candidates to pass the exam provided they meet the standard.

The current passing standard was set in 2021 and will be maintained until the next passing standard study, which occurs approximately every five years. The passing standard is not released. Additional information about the standard setting process is outlined below.

Between passing standard studies, the passing standard is maintained through a psychometrically validated equating process. Statistical equating is used to ensure the performance standard is consistent across multiple administrations of the Oral Examination. Essentially, this means that while the numerical passing score may change with each administration, candidates are not disadvantaged when one administration is more difficult than another. This process accounts for the 4-6-week interval between an exam administration and the release of results.

As a final quality assurance measure before scores are finalized, a key validation process is performed. This process considers the performance of all test items (PMPs), flags any potentially underperforming items and sends them for re-review by subject matter experts to confirm correctness and fairness. If an item is determined to be flawed, it is removed from scoring.

The Standard Setting Process

The cut score or minimum passing standard is set by a group of practicing, board-certified ophthalmologists. The panel members are selected to represent diversity in gender, practice area, time since initial certification, practice environment, and geographical location. Some of the standard setting panelists are examiners whereas others are ABO diplomates who have no knowledge of the Oral Examination other than what they remember from when they took the test themselves. The process is overseen by a psychometrician who is trained in defensible standard-setting practices supported by research.

A standard setting study was held for the March 2021 examination immediately following its administration. The panel discussed and determined a definition of “minimal competence.” In other words, they documented what critical knowledge and skills are necessary for board-certified-level practice. The standard is set such that all examinees theoretically could meet the standard regardless of their peers’ performance; this is referred to as a criterion-referenced standard.

Next, the standard setting panel reviewed the Oral Examination content (the Patient Management Problems, or PMPs). Each group member independently rated the items using the Angoff method(1), which is arguably the most commonly used standard setting method in licensure and certification examinations(2),(3). Specifically, the panelists are asked to conceptualize a candidate who just barely passes and make judgments about this hypothetical candidate’s performance on different PMPs. Eighteen participants (listed below) served on the 2021 standard setting panel, and their ratings and judgments were aggregated and reviewed by the psychometrician. They also participated in independent, holistic judgments about the overall examination difficulty and relevance(4),(5).

These data were presented as a recommendation to a subset of the Board of Directors who reviewed and approved the standard(6). Although ultimately a policy decision, the standard setting process is designed to be as objective as possible by following best practices outlined in the Standards for Educational and Psychological Testing(7) and decades of professional literature in educational measurement and psychometrics.

*After the passing standard was set in 2021, it has been maintained as the standard over time while accounting for small changes in form difficulty. In other words, even if an oral examination is slightly more challenging than a previous version, that difference is psychometrically accounted for so that all examination forms require the same amount of content mastery in order to pass.

Another helpful resource on standard setting developed by the American Board of Emergency Medicine is available here.

References

(1) Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (pp. 508-600). Washington DC: American Council on Education.

(2) Mills, C. N., & Melican, G. J. (1988). Estimating and adjusting cutoff scores: Features of selected methods. Applied Measurement in Education, 1(3), 261-275.

(3) Plake, B. S., & Cizek, G. J. (2012). Variations on a theme: The modified Angoff, extended Angoff, and yes/no standard setting methods. In G. Cizek (Ed.) Setting performance standards (pp. 181-199). New York, NY: Routledge.

(4) Hofstee, W. K. B. (1983). The case for compromise in educational selection and grading. In S. B. Anderson and J. S. Helmick (Eds.), On educational testing (pp. 109-127). San Francisco, CA: Jossey-Bass.

(5) Schnabel, S., Risk, N. (2020, September). Objectifying the Subjective: How the Cognitive Experience of Panelists Shapes the Standard Setting Process. Presentation at the 2020 Association for Test Publishers Innovations in Testing Conference.

(6) Geisinger, K.F. & McCormick, C.M. (2010). Adopting cut scores: Post-standard-setting panel considerations for decision makers. Educational Measurement: Issues and Practice, 29(1), 38–44.

(7) American Educational Research Association, the American Psychological Association, & the National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.