The Secret Handshake of Medical Exams

How Pass Marks Are Really Set

Introduction: The Importance of Standard Setting in Health Professions

The process of determining pass/fail marks for health professions exams, known as "standard setting," is crucial for public safety and maintaining trust in the healthcare system. It establishes the minimum level of competence required for individuals to practice safely and effectively, ensuring they are ready to provide medical care. This process applies to all forms of assessment, from written tests to clinical simulations.

Defining Standard Setting and Its Significance

Standard setting is the procedure for defining the minimum level of competence necessary to pass an exam and practice safely. It draws a line between readiness to practice and the need for further training, guaranteeing that healthcare professionals possess a requisite level of proficiency. The credibility of the healthcare system hinges on rigorous evaluation and the establishment of appropriate standards.

Historical Evolution of Standard Setting

The methods for setting pass marks have evolved significantly over time:

• Early Practices: Historically, pass marks were often arbitrary, with institutions simply selecting a percentage (e.g., 50% or 60%) without a rigorous basis.

• The Flexner Report (Early 1900s): This influential report advocated for more scientific and rigorous standards in medical education, setting the stage for more sophisticated assessment methods.

• Shift from Norm-Referenced to Criterion-Referenced:

  • Norm-Referenced ("Grading on the Curve"): This method compares students to each other, with a certain percentage passing based on their relative performance. However, this approach is criticized because an individual's competence can be dependent on the performance of their peers, which is problematic in a field demanding absolute certainty.

  • Criterion-Referenced: This approach focuses on an individual's performance against a pre-defined set of skills and knowledge. The development of national skill standards (e.g., NCHSE in the 1990s) and specific certifications (e.g., CHES in 1989) marked a significant shift towards ensuring professionals meet core abilities. The goal is to objectively measure actual competence.

    
    

Current Approaches to Standard Setting and Their Challenges

Modern standard setting involves a combination of expert judgment and data analysis, often presenting a complex balancing act. Common methods include:

• Fixed Cut-Off: A fixed percentage (e.g., 50%) is used as the pass mark. This is criticized for being arbitrary and not accounting for the varying importance of different questions or concepts.

• Norm-Referenced Revisited: While statistically refined, this method still relies on cohort performance and may not adequately demonstrate absolute competence.

• Criterion-Referenced Methods (The "Gold Standard"): These aim to define the absolute level of ability required.

  • Angoff Method: Experts estimate the probability that a "just competent" student would answer each question correctly. Challenges include the difficulty of consistently envisioning and applying the profile of a "just competent" student across all items.

  • Borderline Regression Method (BRM): Primarily used for hands-on clinical exams (OSCEs). Examiners rate students' overall performance, and the scores of "borderline" students are used to mathematically set the pass mark. This method is considered robust for skills assessment but still relies on examiner subjectivity.

  • Other Methods: Techniques like Ebel's, Hofstee, and Bookmark are also employed, each with its own strengths and weaknesses.

All chosen methods must be defensible, transparent, and research-backed to be perceived as fair by stakeholders.

"Wicked Problem" Aspects of Standard Setting

Standard setting is considered a "wicked problem" due to its multifaceted nature and lack of simple solutions:

• Human Subjectivity: All methods rely on human judgment, which is inherently subjective and can introduce bias.

• Defining "Minimally Competent": Consistently defining and envisioning the hypothetical "borderline" candidate is challenging, leading to variability in judgments.

• Cut-Score Variability: Different methods or repetitions of the same method can yield significantly different pass marks, raising questions about fairness and reliability.

• Panel Composition: The background and experience of expert panelists can influence the pass mark, making diverse and representative panels crucial.

• Lack of Universal "Best" Method: There is no single, universally agreed-upon superior method for all assessment types.

• Fairness and Accessibility: Standards must comply with legal requirements like the Americans with Disabilities Act (ADA), balancing accommodations with the maintenance of core competencies.

Future Trends in Healthcare Standards

The future of standard setting in health professions education will be shaped by several evolving trends:

• Interprofessional Education (IPE): Increased collaboration among different healthcare disciplines will necessitate standards for teamwork and integrated care.

• Mastery-Based Education: A shift towards competency-based, time-variable education where students progress upon demonstrating mastery, requiring adaptable standard-setting approaches.

• Expanded Competencies: Curricula are broadening to include social determinants of health, mental health, and sustainability, requiring assessments that reflect these wider skill sets.

• Technological Integration:

  • AI and Machine Learning: Potential to assist in personalized learning, simulation generation, and objective standard setting.

  • Virtual and Augmented Reality (VR/AR): Development of standards for assessing performance in immersive virtual training environments.

  • Telemedicine Training: Emergence of new standards for virtual consultations and remote patient care.

• Continuous Vigilance: Regulatory bodies will continue to update frameworks, accreditation, and promote lifelong learning to maintain high professional standards.

Conclusion: The Unsung Heroes of Health Professions Education Quality

Standard setting is a dynamic, critical process at the core of health professions education quality assurance. It is an ongoing effort to balance rigor, fairness, and the evolving demands of medicine, ultimately ensuring that healthcare professionals are truly competent and capable of providing safe and effective care for the collective well-being.