Abstract
Medevi-RoB is a cross-domain benchmark study designed to evaluate large language model performance in automated risk-of-bias assessment of randomized controlled trials, comparing model outputs against consensus ratings from high-quality systematic reviews across a diverse range of specialties.
Multiple leading LLMs, prompting techniques and optimization strategies will be evaluated, including fine-tuning and ensembling. The primary outcome is agreement with expert reviewers, quantified using Cohen’s kappa, F1 score and AUROC. Secondary analyses will explore model confidence, systemic biases, cost-effectiveness and common error patterns.
Background
RoB 2 is the current Cochrane standard for assessing the internal validity of randomized trials, but manual assessment is labour-intensive: studies report mean times of 168.5 to 358 minutes per study. Earlier evaluations of LLMs for risk-of-bias assessment have been limited in scale, reporting only slight to fair agreement with review authors. Medevi-RoB is designed as a large-scale, cross-domain evaluation, and the first in a planned series extending to ROBINS-I, QUADAS-C and AMSTAR.
Objectives
Primary: assess how well state-of-the-art LLMs such as GPT-5, Gemini and DeepSeek reproduce RoB 2 domain-level and overall assessments reported in high-quality systematic reviews.
Secondary:
- Evaluate the impact of prompting strategies on performance.
- Test whether confidence and consistency metrics correlate with accuracy, and the utility of interpretability methods.
- Determine cost-effectiveness and the model size appropriate for most tasks.
- Identify systematic biases by trial and publication characteristics.
- Characterize common model error types against expert consensus.
Methods
Study design and dataset
A cross-sectional diagnostic accuracy study comparing LLM-facilitated RoB 2 assessments, run through Medevi, with expert consensus from Cochrane reviews that used at least two independent reviewers. Parallel-group RCTs with English full text are included; cluster-randomized and crossover trials are excluded. The analysis set is capped at approximately 1,000 RCTs.
Models and prompting strategies
Reasoning, general-purpose and resource-efficient models are compared, alongside supervised fine-tuning, a voting ensemble and an XGBoost baseline on embeddings, with a 70/30 train–test split.
| Strategy | Input | Algorithmic role |
|---|---|---|
| Zero-shot overall | RCT text only | None |
| Per-domain | RCT text, signalling questions | Rules derive overall from domains |
| Per-sub-domain logic | Text, signalling questions, cribsheet | Rules derive domain and overall |
Outcomes
Primary metrics, with 95% CIs, are Cohen’s kappa, agreement, F1 and multi-class AUROC, computed human vs. model, model vs. model and model vs. self. Secondary analyses relate white-box and black-box confidence to accuracy, model systematic under-rating with regression (Benjamini–Hochberg q = 0.05), and classify 100 sampled errors by type.
Openness
Dataset specifications, analysis code, prompt templates and visualizations will be shared in public repositories. DOIs, bibliographic metadata, extracted trial characteristics and derived RoB 2 labels will be released; trial PDFs will not be redistributed.