Areas of interest
Where rigorous evidence meets machine learning.
Medevi is an academic research group building open tools for medicine. Our work sits where clinical science and machine learning meet, and it is organized around four areas.
Context
Six decades of AI in medicine.
Medicine has reasoned with computers since the 1950s and synthesized evidence systematically since the 1970s. Our work builds on both histories.
Scroll sideways · Hover or tap a milestone
- 1959 · Decision supportReasoning foundations of diagnosisLedley and Lusted frame diagnosis in terms of symbolic logic and probability, laying the groundwork for computer-aided diagnosis.Ledley RS, Lusted LB. Science 1959;130(3366):9–21
- 1966 · Machine learningELIZAWeizenbaum's ELIZA, best known for its psychotherapist script DOCTOR, is an early demonstration of natural-language conversation with a computer.Weizenbaum J. Commun ACM 1966;9(1):36–45
- 1972 · Evidence synthesisEffectiveness and EfficiencyArchie Cochrane argues that health care should be guided by evidence from randomised controlled trials.Cochrane AL. Effectiveness and Efficiency. Nuffield Provincial Hospitals Trust, 1972
- 1976 · Decision supportMYCINA rule-based expert system at Stanford recommends antibiotics for bacterial infections: an early clinical decision support system.Shortliffe EH. Computer-Based Medical Consultations: MYCIN. Elsevier, 1976
- 1976 · Evidence synthesisMeta-analysis namedGene Glass coins "meta-analysis" for the statistical analysis of a collection of results from individual studies.Glass GV. Educational Researcher 1976;5(10):3–8
- 1982 · Decision supportINTERNIST-1A diagnostic consultant program for general internal medicine is evaluated on challenging clinicopathological cases.Miller RA, Pople HE, Myers JD. N Engl J Med 1982;307(8):468–76
- 1992 · Evidence synthesisEvidence-based medicineThe Evidence-Based Medicine Working Group, led from McMaster University, describes a new approach to teaching the practice of medicine.Evidence-Based Medicine Working Group. JAMA 1992;268(17):2420–5
- 1993 · Evidence synthesisCochrane CollaborationAn international network forms to prepare, maintain and disseminate systematic reviews of health care interventions.Cochrane, founded 1993
- 1998 · Decision supportCAD for mammographyThe US FDA approves the first computer-aided detection system for screening mammography.US FDA premarket approval, ImageChecker (R2 Technology), 1998
- 2004 · Evidence synthesisGRADEThe GRADE Working Group proposes a system for rating the certainty of evidence and the strength of recommendations.GRADE Working Group. BMJ 2004;328:1490
- 2009 · Evidence synthesisPRISMAA reporting guideline for systematic reviews and meta-analyses, with the now-familiar flow diagram.Moher D et al. BMJ 2009;339:b2535
- 2012 · Machine learningDeep learning breakthroughA deep convolutional network wins the ImageNet challenge by a wide margin, starting the deep learning era.Krizhevsky A, Sutskever I, Hinton GE. NeurIPS 2012
- 2015 · EfficiencyKnowledge distillationLarge models are compressed into smaller "student" models that keep much of their performance at a fraction of the cost.Hinton G, Vinyals O, Dean J. arXiv:1503.02531, 2015
- 2016 · Decision supportDiabetic retinopathy detectionA deep learning algorithm detects referable diabetic retinopathy in retinal photographs with high sensitivity and specificity.Gulshan V et al. JAMA 2016;316(22):2402–10
- 2017 · Machine learningTransformersThe transformer architecture, built on attention alone, becomes the basis of modern large language models.Vaswani A et al. NeurIPS 2017
- 2018 · Decision supportFirst autonomous AI diagnosticThe FDA authorises IDx-DR to detect diabetic retinopathy in primary care without a clinician interpreting the image.Abràmoff MD et al. npj Digit Med 2018;1:39
- 2019 · Trust & biasRacial bias in a health algorithmA widely used algorithm that predicted health needs from costs is shown to underestimate the needs of Black patients.Obermeyer Z et al. Science 2019;366(6464):447–53
- 2021 · Trust & biasSepsis model external validationExternal validation of a widely implemented proprietary sepsis model finds much poorer performance than its developer reported.Wong A et al. JAMA Intern Med 2021;181(8):1065–70
- 2021 · EfficiencyLow-rank adaptationLoRA fine-tunes large language models by training a small number of added parameters, cutting the compute needed for adaptation.Hu EJ et al. arXiv:2106.09685, 2021
- 2022 · Machine learningChatGPTOpenAI releases ChatGPT in November 2022, bringing large language models into everyday use, including by clinicians.OpenAI, November 2022
- 2023 · Machine learningLLMs encode clinical knowledgeFlan-PaLM passes the mark on US Medical Licensing Examination–style questions; the instruction-tuned Med-PaLM narrows, but does not close, the gap with clinicians.Singhal K et al. Nature 2023;620:172–80
- 2024 · Trust & biasTRIPOD+AIUpdated reporting guidance for clinical prediction models that use regression or machine learning methods.Collins GS et al. BMJ 2024;385:e078378
Sources for the timeline ShowHide
- 1959Ledley RS, Lusted LB. Science 1959;130(3366):9–21
- 1966Weizenbaum J. Commun ACM 1966;9(1):36–45
- 1972Cochrane AL. Effectiveness and Efficiency. Nuffield Provincial Hospitals Trust, 1972
- 1976Shortliffe EH. Computer-Based Medical Consultations: MYCIN. Elsevier, 1976
- 1976Glass GV. Educational Researcher 1976;5(10):3–8
- 1982Miller RA, Pople HE, Myers JD. N Engl J Med 1982;307(8):468–76
- 1992Evidence-Based Medicine Working Group. JAMA 1992;268(17):2420–5
- 1993Cochrane, founded 1993
- 1998US FDA premarket approval, ImageChecker (R2 Technology), 1998
- 2004GRADE Working Group. BMJ 2004;328:1490
- 2009Moher D et al. BMJ 2009;339:b2535
- 2012Krizhevsky A, Sutskever I, Hinton GE. NeurIPS 2012
- 2015Hinton G, Vinyals O, Dean J. arXiv:1503.02531, 2015
- 2016Gulshan V et al. JAMA 2016;316(22):2402–10
- 2017Vaswani A et al. NeurIPS 2017
- 2018Abràmoff MD et al. npj Digit Med 2018;1:39
- 2019Obermeyer Z et al. Science 2019;366(6464):447–53
- 2021Wong A et al. JAMA Intern Med 2021;181(8):1065–70
- 2021Hu EJ et al. arXiv:2106.09685, 2021
- 2022OpenAI, November 2022
- 2023Singhal K et al. Nature 2023;620:172–80
- 2024Collins GS et al. BMJ 2024;385:e078378
01 · Area of interest
Evidence synthesis
We are dedicated to accelerating the pace of rigorous evidence synthesis, and envision a future where clinicians always have the latest evidence at their fingertips for decision-making.
- 67 weeks
- Mean time to complete and publish a systematic review registered in PROSPERO.
- Borah R et al. BMJ Open 2017;7:e012545
- 23%
- Of systematic reviews showed a signal that they needed updating within two years.
- Shojania KG et al. Ann Intern Med 2007;147(4):224–33
Drag the slider or press Play. Each new trial narrows the pooled estimate. A living review would update the moment a trial is published, not years later.
02 · Area of interest
Clinical decision support systems
Validated clinical decision scores and ML tools are increasingly used in medicine. We seek to facilitate their implementation in real-world systems.
A score is only as good as its implementation. When a widely used proprietary sepsis model was externally validated, it performed far worse than reported (Wong et al., 2021). We build open, versioned, cited implementations that any system can call and anyone can check.
Open risk-score calculators, coming soon →- 01Derived and validatedPublished score with external validation
- 02Open specificationVersioned, cited, machine-readable
- 03InterfacesMCP tools and EMR integration
- 04Point of careSame result, wherever it is used
- Age
- 70+1
- Sex
- Female+1
- Congestive heart failure
- No0
- Hypertension
- Yes+1
- Stroke, TIA or thromboembolism
- No0
- Vascular disease
- No0
- Diabetes mellitus
- Yes+1
{
"id": "cha2ds2_vasc",
"version": "1.0.0",
"purpose": "Stroke risk in atrial fibrillation",
"source": {
"citation": "Lip GYH et al. Chest 2010;137(2):263–72",
"doi": "10.1378/chest.09-1584"
},
"inputs": {
"age": { "type": "integer", "points": { "65-74": 1, ">=75": 2 } },
"sex": { "enum": ["male", "female"], "points": { "female": 1 } },
"chf": { "type": "boolean", "points": 1 },
"hypertension": { "type": "boolean", "points": 1 },
"stroke_tia_te": { "type": "boolean", "points": 2 },
"vascular_disease": { "type": "boolean", "points": 1 },
"diabetes": { "type": "boolean", "points": 1 }
},
"output": { "type": "integer", "range": [0, 9] }
}// request
{
"jsonrpc": "2.0", "id": 7,
"method": "tools/call",
"params": {
"name": "cha2ds2_vasc",
"arguments": {
"age": 70, "sex": "female",
"hypertension": true, "diabetes": true
}
}
}
// response
{
"jsonrpc": "2.0", "id": 7,
"result": {
"structuredContent": {
"score": 4, "max": 9,
"source": "doi:10.1378/chest.09-1584",
"spec_version": "1.0.0"
}
}
}Draft schema, for illustration · not for clinical use
03 · Area of interest
Trustworthiness, bias and interpretability
We are interested in creating open models which clinicians can trust, understand, and use reliably, while being cognizant of potential biases inherent to our models.
- Source-linkedEvery judgement points to the sentence that supports it.
- MeasuredAgreement with expert reviewers, with confidence intervals.
- AuditedPerformance reported by subgroup, and errors described.
- OpenModels, prompts and benchmarks others can rerun.
Methods. Adults with community-acquired pneumonia were enrolled at six centres. Participants were randomized 1:1 using a computer-generated sequence, with allocation concealed in sequentially numbered, opaque, sealed envelopes. Owing to the nature of the intervention, participants and treating clinicians were aware of group assignment. The primary outcome, clinical cure at day 14, was assessed by investigators blinded to allocation. Results. Outcome data were available for 412 of 420 randomized participants (98%). The trial was registered after enrolment began, and the statistical analysis plan was not published.
- Overall0.84
- Age ≥ 750.71
- Female0.83
- Male0.85
- Rural sites0.74
A model can look strong overall and still underperform for some patients. We report results by subgroup, with intervals, before anyone relies on them.
04 · Area of interest
Model efficiency
Clinical data and tasks alike face redundancy. We aim to make clinical implementations of machine intelligence more parsimonious.
Fewer inputs to collect, smaller models to host, lower cost per task: parsimony is what lets a tool run inside a hospital's own infrastructure, not just in a lab.
- $0.0034
- per paper for single-shot RoB 2 assessment with the open gpt-oss-120b model.
- Medevi research →
- 29–57 s
- per paper for automated MERSQI and NOS-E appraisal, against 65–130 minutes for a human reviewer.
- Medevi research →
Drag the slider. In many clinical prediction tasks, the first few inputs carry most of the signal; the rest add cost and burden for little gain.
Collaborate