At ICAM 2026 the Medevi team presented a benchmark of large language models on the two quality instruments most used in medical education research: the Medical Education Research Study Quality Instrument (MERSQI) and the Newcastle–Ottawa Scale–Education (NOS-E).
Why automate critical appraisal?
Risk-of-bias assessments are essential for trustworthy evidence synthesis, but they are slow, variable and costly: a systematic review costs $50,000–$140,000 and takes 6–18 months to complete. Automation tools such as Elicit and RobotReviewer are increasingly adopted, yet they are limited in capability or lack benchmarks for appraisal tasks. Consumer chatbots are widely used, but they are not validated for research and can hallucinate critical information.
What we did
We assembled 139 peer-reviewed articles with published MERSQI and NOS-E scores from five systematic reviews, covering retrieval practice, surgical skills, pain management education and virtual reality simulation. Using Cook and Reed’s (2015) operational definitions, we ran GPT-5.2, GPT-5 mini and GPT-5 nano through Medevi in zero-shot and per-domain chain-of-thought modes, and measured agreement and intraclass correlation against the human ratings.
Key findings
- LLM-assisted extraction achieved substantial agreement with human reviewers on total MERSQI score (ICC 0.755; 95% CI 0.704–0.801), and fair agreement on total NOS-E score (ICC 0.345; 95% CI 0.268–0.424).
- Domain-level exact agreement ranged from 35.3% to 94.9% for MERSQI and 34.9% to 78.5% for NOS-E. Domains that need specialist judgement, such as validity evidence, were hardest.
- Zero-shot evaluation with current long-context models was non-inferior to per-domain prompting on MERSQI.
- Medevi took 29–57 seconds and $0.10–$0.57 per paper, against 65–130 minutes and $21.67–$43.33 for a human reviewer.
“With robust benchmarks, we can move from experimental use of AI to robust use and evaluation in evidence synthesis.”
What’s next
We are now testing Medevi on title/abstract and full-text screening for medical education reviews, studying bias in model outputs and how useful reviewers find the explanations, and extending the benchmark to RoB 2, ROBINS-I, QUADAS-2 and PROBAST.