The Open Medical-LLM Leaderboard provides a framework for comparing AI models on medical questions, but its current interactive rankings could not be verified at an October 12, 2026 research check. The official Hugging Face application displayed a runtime error, while public source files and historical results remained accessible. Those artifacts do not establish a current best-performing model. 1 2 3
This is a historical benchmarking initiative, not a verified new launch. Hugging Face’s announcement is dated April 19, 2024. Open Life Science AI hosts the project on Hugging Face, and the announcement acknowledges Hugging Face contributors and GPU support from the University of Edinburgh. 4
The distinction matters for healthcare organizations assessing AI systems: the maintainers describe the leaderboard as a research evaluation tool, not a mechanism for clinical deployment or approval. Its scores measure answers to benchmark questions—not whether a model can safely manage a clinical workflow. 5
What the leaderboard measures
The documented backend uses EleutherAI’s Language Model Evaluation Harness. Project source configures nine tasks with answer-accuracy metrics and a default of zero few-shot examples, meaning the default setup does not provide worked examples before the question. That default does not describe every imported result. 5
The documented coverage includes:
- MedQA: 1,273 U.S. medical licensing examination questions.
- MedMCQA: Questions from Indian medical entrance examinations, with conflicting counts in project materials.
- PubMedQA: 500 questions requiring a yes, no or maybe answer using a biomedical abstract as context.
- Six MMLU subsets: Clinical knowledge, medical genetics, anatomy, professional medicine, college biology and college medicine. 4 5
The MedMCQA discrepancy remains unresolved. Hugging Face’s announcement describes 6,100 test questions, while the leaderboard documentation lists 4,183 questions. These figures cannot be assumed to represent the same scored denominator. Confirming the evaluated split requires checking execution configurations and result artifacts. 4 5
The model comparisons have a protocol caveat
Hugging Face’s 2024 announcement described GPT-4-base and Med-PaLM-2 as strong performers, while roughly seven-billion-parameter Starling and Hermes models were competitive on selected tasks. Those are the announcement authors’ historical performance descriptions, not verified 2026 rankings. 4
More importantly, the documentation says GPT-4 and Med-PaLM-2 results were imported from papers. It explicitly includes five-shot Med-PaLM-2 results alongside otherwise described zero-shot results, while Gemini results come from a separate ClinicalNLP study. The comparisons therefore should not be presented as uniformly rerun experiments under identical conditions. 5
Public code and model-specific result files provide a starting point for auditing these differences. Their availability does not, by itself, establish that every row can be reproduced or compared under one consistent protocol. 2 3 5
Medical knowledge is not clinical readiness
Answer accuracy is a narrower measure than clinical safety. On its own, it does not establish whether a model expresses uncertainty appropriately, avoids the most consequential errors, performs equitably or improves patient outcomes. That is an interpretation of the benchmark’s scope, not an additional finding produced by its scores. 5 6
Independent research illustrates why broader testing matters. A July 2024 Nature Medicine study evaluated clinical decision-making using 2,400 patient cases across four abdominal conditions. Researchers found shortcomings in the tested models’ information gathering, guideline adherence and diagnosis relative to physicians. The findings concern those models and conditions; they should not be generalized to every model available in 2026. 6
The RABBITS study, published in EMNLP 2024 Findings, identified another limitation: substituting equivalent brand and generic drug names reduced tested models’ performance on MedQA and MedMCQA. Researchers also found benchmark material in public pretraining corpora, but cautioned that those corpora were proxies—not confirmed training datasets for every evaluated model. The results support robustness and contamination checks, not a blanket claim that all leaderboard entries are compromised. 7
Neither study independently replicated the Open Medical-LLM Leaderboard’s rankings. They provide context for interpreting what medical question-answering scores can—and cannot—show. 6 7
Later benchmarks ask broader questions
Other projects illustrate different evaluation approaches. OpenAI said its HealthBench benchmark, announced May 12, 2025, uses 5,000 realistic health conversations and physician-created grading rubrics, including multi-turn and multilingual scenarios. Those scenarios are simulated; HealthBench is not a patient-outcomes trial. 8
Stanford’s April 8, 2025 description of MedHELM covered more than 120 scenarios across decision support, clinical documentation, patient communication, research assistance and administration. These broader task categories make it a relevant comparator, but its measurements should not be numerically equated with the Open Medical-LLM Leaderboard’s question-answering accuracy. Neither project is established here as an interchangeable scoreboard or proven replacement. 9 8
Availability and evaluation freshness remain unresolved
The application error is a point-in-time observation, not evidence of permanent discontinuation. A separate community post dated November 15, 2024, reported slow processing and an inability to submit a model. That post records one user’s experience, not a maintainer-confirmed diagnosis of a system-wide outage. Together, these observations do not establish continuous downtime or its cause. 1 10
The latest visible main-branch commit was dated September 19, 2025, and created a Dockerfile. That is evidence of code maintenance, not refreshed benchmark evaluations. Likewise, the accessible results repository contains model-specific JSON artifacts, including numerous filenames dated in 2024, but historical records do not establish current evaluation activity. 11 3
The remaining reporting questions are concrete: Who currently maintains the evaluation infrastructure? When did it last complete a new model evaluation? Which dataset splits, harness revisions, prompts, chat templates and inference settings underpin each row? Can imported results be reproduced consistently, and what checks address contamination and statistical uncertainty? The supplied research does not settle those questions.
For researchers, the project’s public artifacts remain an audit starting point. For healthcare organizations, the defensible use is narrower: knowledge scores can inform initial screening, but they should not substitute for workflow-specific evidence of clinical behavior or patient benefit. 2 3 6