The University of Texas at Austin wordmark
Tutorial
NeuroMetaBench & AutoNIMA
Benchmarking and automating coordinate-based neuroimaging meta-analysis
Alejandro de la Vega
October 22, 2026
Today
Two projects, one workflow
Search
PubMed query
Screen
Abstracts, then full text
Extract
Coordinate tables
Select
Analyses for each contrast
Meta-analyze
MKDA or ALE map
Part 1
NeuroMetaBench
The answer key. Expert meta-analyses rebuilt from their curation records, so automated tools can be scored against what experts did.
Part 2
AutoNIMA
The tool. An LLM-assisted pipeline that runs every stage above from one YAML config, and is scored on NeuroMetaBench.
Part 1
NeuroMetaBench
A shared answer key for automated meta-analysis tools
NeuroMetaBench · Overview
Why a benchmark?
Automated meta-analysis tools are widely used, but how closely they recover expert-curated syntheses hasn’t been established.
NeuroMetaBench links expert curation records across the stages of a systematic review, in three tiers:
Tier A · Protocol
22 registered, 14 complete protocols
Review question, search, eligibility criteria
Tier B · Inclusion
17 projects, 1,618 study inclusions
Included-study lists, plus 3 candidate pools
Tier C · Analysis
9 projects, 38 targets
Analyses, coordinates, target assignments, reference maps
NeuroMetaBench · Overview
Four linked records
Meta-analysis
One published project: its criteria, curation records and targets
Study
A publication in that project, keyed by PMID, with DOI as backup
Analysis
One reported result: a set of coordinates and its metadata
Target
A question or contrast, defined by the analyses experts assigned to it
  • An analysis can feed several targets: assignments aren’t exclusive.
  • Records are stored in NiMADS, the Neurosynth Compose format, so NiMARE reads them directly.
  • Each target gets a reference map built from its expert-selected analyses.
Source: NeuroMetaBench draft, Tables 1 and 2
NeuroMetaBench · Tier C
Nine projects, 38 targets
Project
Modality
Studies
Analyses
Coordinates
Targets
Analyses per study
Cue reactivity
fMRI
191
275
3,270
6
1.44
Decision making
fMRI
90
144
1,223
3
1.60
Dementia
fMRI, structural
29
31
1,410
4
1.07
Emotion regulation
fMRI
90
248
2,829
4
2.76
Executive function
fMRI
178
271
3,076
4
1.52
Problem solving
fMRI
129
282
3,166
5
2.19
Social cognition
fMRI
238
856
7,185
5
3.60
VBM of PTSD
Structural
22
25
159
1
1.14
VBM of substance use
Structural
80
87
673
6
1.09
Total
1,047
2,219
22,991
38
2.12
NeuroMetaBench · Overview
Analyses, not articles
2.12
analyses per study on average (range 1–13)
55 of 59
studies behind emotion regulation “maintain” also feed “decrease”, yet the two share no analyses
Sibling targets can share studies but not analyses, so article-level inclusion can’t tell them apart.
Scatter plot of study overlap versus analysis overlap for 35 pairs of sibling targets. Dementia pairs lie on the diagonal; emotion regulation pairs share up to 61% of studies but no analyses.
Source: NeuroMetaBench draft, Fig. 1 · Jaccard overlap of studies vs analyses for 35 sibling-target pairs
NeuroMetaBench · Overview
How tools are scored
Every reference is re-estimated the same way: MKDA in NiMARE 0.21, 10 mm kernel, analyses weighted equally, FDR q = 0.05, on the 2 mm MNI152 grid.
Raw r²
Squared voxelwise correlation between a tool’s map and its intended reference
Identification
Share of the other references that match a tool’s map less well than its own. Chance is 0.5.
Sibling discrimination
Identification against the target’s own project only: the hard version
Split-half reliability
Split the studies in half ten times and correlate the two maps. 0.6 or more counts as reliable.
31 evaluable references: four substance-use maps with no significant voxels are excluded
NeuroMetaBench · How to contribute
Add your meta-analysis
Ran a coordinate-based meta-analysis and kept the record of which analyses went into each map? That record is the contribution. The analyses you rejected help too.
1
Register it
One row in data/meta_datasets.csv: search, criteria, counts, contact
2
List included studies
One row per study in data/included_studies.csv; screening decisions are optional
3
Add coordinates
One Sleuth text file per target, or a Neurosynth Compose project
4
Convert to NiMADS
NiMARE converts Sleuth files; fuzzy matching proposes PMIDs and you confirm each
5
Build reference maps
MKDA in NiMARE, the same recipe as every other target
6
Open a pull request
From your fork to neurostuff/neurometabench
The repository is private until the public release, so ask for access. Corrections to existing records are welcome too.
NeuroMetaBench · How to contribute
The files you edit
Spreadsheet view of data/meta_datasets.csv: one row per meta-analysis with status, PMID, year, topic, first author, number of studies, method, modality, search and inclusion criteria. The VBM of PTSD row is highlighted.
Spreadsheet view of data/included_studies.csv with columns meta_pmid, study_pmid and doi, showing rows for the PTSD meta-analysis.Text editor view of a Sleuth file: a Reference=MNI header, then a study label with contrast, its subject count, and x y z coordinates.
1 · one row per meta-analysis
2 · one row per included study, keyed by PMID
3 · one Sleuth file per target: each study’s label, sample size and x y z peaks
NeuroMetaBench · How to contribute
The same steps as commands
# 1–3  add your rows and files
data/meta_datasets.csv
data/included_studies.csv
raw/meta-datasets/<N>. <Topic> - <Author>/<target>.txt   # Sleuth, // Reference=MNI
 
# 4  convert to NiMADS, then link author–year labels to PMIDs
python scripts/convert_sleuth_to_nimads.py --project <project>
python scripts/fuzzy_match_analysis_to_pmid.py --project <project> \
    --included-studies data/included_studies.csv \
    --output-dir raw/fuzzy_match_study_to_pmcid --threshold 0.85
# review the matches, then rerun the conversion with --fuzzy-map-dir
 
# 5  build the reference maps
pixi run -e nimare021 python scripts/run_meta_analyses.py --projects <project>
Heads-up: the conversion script expects a patched NiMARE installed in editable mode. Environments come from pixi (Linux and Windows only).
NeuroMetaBench · Results
Reference maps cluster by project
0.50
mean correlation between reference maps from the same project
0.20
between maps from different projects
Executive function, problem solving and perceptual decisions sit together. Dementia sits apart from the task-based maps.
Multidimensional scaling plot of 26 reference maps, colored by project. Targets from the same project sit together; emotion regulation maintain sits far from its siblings.
Source: NeuroMetaBench draft, Fig. 2 · multidimensional scaling of 26 reference maps
NeuroMetaBench · Results
Some references are noisier than others
Split-half reliability of each reference map: how well two halves of its studies reproduce each other
reliable ≥ 0.6
Risky decisions 0.17
VBM: PTSD 0.25, alcohol 0.27
ER decrease 0.84
Executive function 0.72–0.90
0
0.4
1
11 of 31
evaluable references fall below 0.5. Small samples explain part: halves of 7 to 20 studies.
4
substance-use references had no significant voxels and near-zero reliability, so they’re excluded
Source: NeuroMetaBench draft, Fig. S1 · ten random splits per target, Spearman–Brown corrected
NeuroMetaBench · Results
Tools get the domain, not the contrast
Three dot-and-interval panels for NeuroQuery, NeuroVLM, Neurosynth association and Neurosynth uniformity: raw r-squared, identification and sibling discrimination.
a · Raw r²: 0.07 to 0.28. Neurosynth uniformity is highest, being closest to a conventional CBMA.
b · Identification: 0.74 to 0.83, above chance (0.50). No two methods differ.
c · Sibling discrimination: 0.57 to 0.65. Every interval includes chance, even for reliable references.
Source: NeuroMetaBench draft, Fig. 3 · 31 evaluable targets (30 with a sibling in c) · 95% CIs from a bootstrap over projects
NeuroMetaBench · Results
Reliable references can still be missed
Shared failures
Identification ranges from 0.27 to 1.00, and the methods agree on which targets are hard (Kendall’s W = 0.70).
ER · decrease
Reliability 0.84. NeuroQuery’s map correlates 0.07 with it, and 0.31 with working memory.
Cue · natural reward
Reliability 0.72. NeuroQuery’s map correlates 0.21 with it, and 0.34 with drug cues, though the two share only 3 of 160 studies.
Panel a: identification scores for 14 reliable references by four methods, with emotion regulation decrease below chance. Panels b and c: glass-brain maps of each reference, NeuroQuery's map, and the reference NeuroQuery's map matches instead.
Source: NeuroMetaBench draft, Fig. 4 · the 14 references with split-half reliability ≥ 0.6, scored among 26
Part 2
AutoNIMA
LLM-guided selection of articles and analyses, from a PubMed query to a meta-analysis map
AutoNIMA · Overview
Seven stages, four with an LLM
1
Deterministic
Search
PubMed through the Entrez API
2
LLM
Abstract screening
Lenient criteria on titles and abstracts
3
Deterministic
Full-text retrieval
PubMed Central through pubget, or article HTML you supply
4
LLM
Full-text screening
The complete eligibility criteria
5
LLM
Coordinate parsing
Splits tables into analyses by contrast, group and direction
6
LLM
Analysis selection
Assigns each analysis to the target contrasts it fits
7
Deterministic
Meta-analysis
NiMADS export, then NiMARE: MKDA, ALE or KDA
Auditable and cached
Every LLM call returns a decision per criterion with a rationale. Results are cached per stage and per study.
AutoNIMA · Results
Scored on NeuroMetaBench
Nine published expert meta-analyses and 32 target contrasts, 28 of them compared at the map level; gpt-5-mini for every LLM stage
0.557
mean r² with the expert maps, against 0.428 for a search-only baseline
26 of 28
contrasts where AutoNIMA’s map is closer to the expert map
91.4%
of expert-curated analyses recovered by LLM parsing, against 33.3% with one analysis per table
$194
estimated model cost for all nine projects: a median $0.085 per study in a final map
Source: AutoNIMA draft manuscript · dementia is left out of map comparisons because its reference pooled coordinates across studies
AutoNIMA · Results
Screening: precision up, recall kept
Line charts by project across four stages. Left: precision against the gold standard rises from about 0.10 at search to 0.47 at analysis selection. Right: recall of attainable gold studies falls gently from 1.0 to 0.83.
47%
mean precision after analysis selection, up from 10.3% after search
83%
of the attainable expert-included studies still kept at the end
65.7%
final precision when starting from the authors’ own candidate lists (3 projects)
Source: AutoNIMA draft manuscript, Fig. 2 and Fig. S2 · recall counts only studies whose full text could be retrieved
AutoNIMA · Results
Parsing and assigning analyses
Left: per-project percentage of expert analyses recovered, tables-only versus LLM parsing. Right: true-positive against false-positive rate for assigning analyses to contrasts, with every project above the diagonal.
91.4%
of expert-curated analyses recovered, against 33.3% treating each table as one analysis
0.68
mean TPR − FPR when assigning analyses to contrasts (chance is 0)
1.8–5.6×
the precision of random selection, depending on the project
Source: AutoNIMA draft manuscript, Fig. 3
AutoNIMA · Results
Maps: 26 of 28 contrasts improve
Left: scatter of AutoNIMA r-squared against search-only baseline r-squared for 28 contrasts, most above the identity line. Right: bars of the r-squared gain per contrast, two slightly negative, mean +0.129.
+0.129
mean gain in r² over search-only (95% CI +0.061 to +0.223)
Sign test P = 3.0 × 10⁻⁶
Gains often came with fewer pooled analyses, so more data isn’t the explanation.
Source: AutoNIMA draft manuscript, Fig. 4 · “Pipeline” in the figure is AutoNIMA
AutoNIMA · Results
The gain comes from selecting analyses
Two scatter plots. Left: article screening only against search-only, points on the diagonal, mean gain +0.008. Right: analysis selection only against pooling all analyses, most points above the diagonal, mean gain +0.113.
+0.008
mean Δr² from article screening alone, on top of already-targeted searches
+0.113
mean Δr² from selecting analyses within the expert-included articles, better in 19 of 28
Selected analyses also beat equally sized random subsets in 26 of 28 contrasts.
Source: AutoNIMA draft manuscript, Fig. 6 and Fig. S3
AutoNIMA · Results
Separating sibling contrasts
Inflated cortical surfaces for three emotion regulation contrasts: increase, maintain and reappraisal. Expert maps on top, AutoNIMA maps below, and a single search-only baseline map on the right.
Search-only pooling gives one map for all three emotion regulation contrasts. Selecting analyses recovers each one. Dice overlap: AutoNIMA 0.53, 0.49 and 0.72; search-only 0.11, 0.20 and 0.44.
Source: AutoNIMA draft manuscript, Fig. 5 · “Pipeline” in the figure is AutoNIMA · z > 1.96
AutoNIMA · Supplement
Agent workflows land close to AutoNIMA
Dot plot by project for three skills-based agent workflows and AutoNIMA. Left: screening F1 against expert inclusions. Right: mean map correlation with the expert map. Means of nine projects cluster near AutoNIMA on both.
0.42–0.46
mean screening F1 for the three agent workflows, against 0.44 for AutoNIMA
0.68–0.71
mean map correlation with the expert maps, against 0.71 for AutoNIMA
Project by project they spread wider: on dementia, from 0.44 to 0.60.
Source: AutoNIMA draft manuscript, Fig. S6 · map r, not r², over all nine projects including dementia
AutoNIMA · Results
Read the results with care
Not a held-out test
Criteria were revised using benchmark feedback, so the estimates are optimistic.
Full text is the bottleneck
About a third of studies were retrievable from open access, and a third of included articles yielded no analyses.
Uneven gains
Smaller for decision making and executive function.
Criteria matter most
Fixing omissions in two projects’ criteria gained +0.221 r²; LLM refinement added +0.031.
AutoNIMA · How to use
Install
Python
3.10 or newer (tested on 3.10 to 3.13)
API key
OPENAI_API_KEY, needed from abstract screening on. Search alone is free.
NCBI email
Goes in the config’s search.email field
Disk
About 1 GB for the environment
# a fresh environment, Python 3.10 or newer
python3 -m venv .venv
source .venv/bin/activate
 
git clone https://github.com/neurostuff/autonima.git
cd autonima
pip install -e ".[llm,meta,ui]"
cd ..
 
autonima --help
export OPENAI_API_KEY=...
Tested from scratch on Python 3.10, 3.11, 3.12 and 3.13 · keep the quotes: zsh on macOS needs them
AutoNIMA · How to use
API keys and accounts
OPENAI_API_KEY · required
Powers the four LLM stages. Create one at platform.openai.com under API keys; the account needs credit.
NCBI email · required
PubMed wants a contact address: search.email in the config.
NCBI_API_KEY · optional
Faster PubMed downloads. From your NCBI account settings.
OPENAI_API_GATEWAY · optional
Base URL of any OpenAI-compatible endpoint, such as a campus gateway or a local model server.
AutoNIMA web UI settings page, Configuration and Secrets: preferred model names, then fields for OPENAI_API_KEY (masked), OPENAI_API_GATEWAY, PUBGET_API_KEY and NCBI_EMAIL.
# command line: set it in your shell
export OPENAI_API_KEY="sk-..."
# web UI: gear icon, Configuration & Secrets
# saved to ~/.autonima.env, used by UI runs
# never put keys in the YAML or in git
AutoNIMA · How to use
One YAML file per review
Which papers
search:
  query: 'VBM[tiab] AND PTSD[tiab]'
  email: you@university.edu
retrieval:
  sources: [pubget]
screening:
  abstract:
    model: gpt-5-mini-2025-08-07
    objective: Grey matter changes in PTSD
    inclusion_criteria:
      - MRI grey matter, PTSD vs controls
  fulltext:  # same keys, stricter
    exclusion_criteria:
      - Only region-of-interest analyses
Which analyses
parsing:
  parse_coordinates: true
annotation:
  enabled: true
  annotations:
    - name: decreased_gm
      inclusion_criteria:
        - Whole brain, patients < controls
output:
  nimads: true
Add one entry under annotations per target contrast. Each becomes its own meta-analysis map.
Trimmed from the VBM of PTSD project config · autonima create-sample-config writes a complete starting file
AutoNIMA · How to use
Run it, cheapest step first
autonima create-sample-config > config.yaml
autonima validate config.yaml
autonima run-search config.yaml       # free: PubMed only, sizes your query
autonima run-abstract config.yaml     # cheap pilot of your criteria
autonima run config.yaml my_review    # the full pipeline
autonima meta my_review/outputs      # NiMARE maps: MKDA by default
autonima ui --workspace .            # the same, in your browser
What you get
  • NiMADS files, ready for NiMARE
  • Interactive reports of every decision and why
  • One map per target contrast
  • missing_fulltexts.csv: papers to fetch yourself
The web UI
A local app with live progress and logs, cancel and clone runs, and a results browser.
Screenshots on the next three slides.
AutoNIMA · Web UI
The same pipeline, in a browser
autonima ui --workspace .
# open the exact URL it prints
  • Write the config as forms, or import a YAML you have
  • Count PubMed hits before screening anything
  • Launch runs with live progress, logs and cancel
  • Browse the meta-analysis maps when a run finishes
AutoNIMA web UI, Find studies step for the VBM of PTSD project: PubMed search query, date range, and buttons to open the search in PubMed or count results.
Screenshots: the VBM of PTSD project imported into a fresh workspace
AutoNIMA · Web UI
Criteria as forms, not YAML
Screening
Numbered inclusion and exclusion criteria, checked one by one
Screening step of the web UI: abstract screening objective, inclusion criteria I1 and I2, and exclusion criteria E1 to E3 for the PTSD project.
Parsing + annotation
One named rule set per target contrast, here increased and decreased grey matter
Annotation-specific rules in the web UI: Annotation 1 named increased_gm and Annotation 2 named decreased_gm, each with its own inclusion criterion.
The forms edit the same YAML config the command line uses
AutoNIMA · Web UI
Check the plan, then start
Run Screening page: a Start Screening button and worker count, then Planned work cards for each stage showing whether verified results are reused or recomputed.
Before anything runs, the plan shows which stages reuse verified results and which recompute. Start it, then watch progress and logs; the meta-analysis step unlocks when screening finishes.
Questions?
AutoNIMA
Docs: neurostuff.github.io/autonima
Code: github.com/neurostuff/autonima
Configs and results: github.com/neurostuff/autonima-results
NeuroMetaBench
Code: github.com/neurostuff/neurometabench, private until the public release
Have a meta-analysis to contribute? Come talk to me.
Alejandro de la Vega · The University of Texas at Austin