Medical BenchDossier01
Original 2018 paper and archive
VQA-RAD ↗
Count images, question records and question variants separately.
UnitOne image-question pairMeasureOriginal manual simple accuracy
Medical visual question answering
The image, the question, the split.
Independent analysis of VQA-RAD and SLAKE: original dataset counts, question variants, image splits, visual grounding, knowledge access and paper-reported results.
Keep images and question forms separate
Original question forms
3,515 question forms in the listed groups.
Original image partitions
642 images in the listed groups.
The two bars have different units. VQA-RAD lists question forms; SLAKE lists image partitions. Read the split and release caveats →
01 /
Counting units
Distinguish images, question records and related question forms.
02 /
Task evidence
Separate image understanding from external-knowledge access.
03 /
Comparison policy
Keep release, language, split and answer judging visible.
Our analytical question
A medical VQA result depends on which images, question forms and answer rules were evaluated. We examine VQA-RAD’s clinician-generated questions and SLAKE’s visual and knowledge-based tasks, keeping paper counts separate from cleaned releases. Our original coverage map and guides explain what those choices support. Selected historical results are credited to the original papers; no new model runs are claimed.
Read the original evidence closely
The benchmark, unpacked.
Dossier02
2021 paper; cleaned SLAKE 1.0 tracked separately
SLAKE ↗
A visual answer and a knowledge answer test different things.
UnitOne image-question pairMeasureAnswer accuracy by task and language
An original analytical tool
Inspect the visual-question task
Compare published question forms, answer formats, evidence requirements and languages. These are qualitative task annotations, not new model scores.
8 of 8 evidence entries shown
- Input
- Image and naturally phrased question
- Output
- Short visual answer
- Measure
- Declared manual or later scoring rule
- Input
- Same image with related question wording
- Output
- Two answers
- Measure
- Correctness plus separately designed pair consistency
- Input
- Image and closed question
- Output
- Restricted answer
- Measure
- Original simple accuracy
- Input
- Image and open question
- Output
- Free-text answer
- Measure
- Original simple accuracy
- Input
- Image and visual question
- Output
- Answer class
- Measure
- Accuracy
- Input
- Image/question and permitted knowledge graph
- Output
- Answer class
- Measure
- Accuracy
- Input
- English questions in a named release
- Output
- Answer class
- Measure
- Accuracy
- Input
- Chinese questions in a named release
- Output
- Answer class
- Measure
- Accuracy
This is our independent analytical map of published tasks. It does not execute an evaluation, predict a model’s performance or establish clinical benefit. [1][2][3][4][5][6]
Original analysis / methods and interpretation
What the score leaves unsaid.
Benchmark analysis
VQA-RAD: reconcile 3,515 questions with 2,248 records
An original accounting of question forms, image reuse and the original VQA-RAD evaluation.
Benchmark analysis
SLAKE: freeze the release, language and knowledge condition
Understand the original image split and why cleaned releases require their own denominators.
Benchmark analysis
What VQA-RAD and SLAKE can tell you about visual grounding
A task-specific audit of images, answer shortcuts and scoring rules in two medical VQA benchmarks.
Questions, answered
Read the result in context.
Specific tasks. Stated conditions.
Inspect every source.
Why does VQA-RAD have both 3,515 and 2,248 counts?
The paper’s headline includes free-form, rephrased and framed question forms. Its Data Records section describes 2,248 elements corresponding to the free-form and rephrased categories.
Does a held-out VQA-RAD question imply a new image?
No such assumption follows from a question-level split. Audit image identifiers and linked question forms in the chosen release.
Does SLAKE 1.0 exactly match the paper?
The authors’ project page explicitly warns that the cleaned release may differ. State the release and count records after applying the language filter.
Do these sites reproduce the official leaderboards?
No. We provide independent analysis and selected historical paper-reported measurements, with their original conditions and limitations.
Working tool / saved on this device
Prepare a benchmark comparison brief
Use this secondary checklist to document a run or literature comparison after inspecting the named benchmark conditions. Completion records documentation, not performance.
Evidence you can inspect. Benchmark dossiers distinguish published facts from our interpretation, with source versions and access notes attached.
Download the evidence ↗