Tutorial

Note

For this tutorial we are using a slimmed down replication of Neurobiological impact of nicotinic acetylcholine receptor agonists: An ALE meta-analysis of pharmacological neuroimaging studies.

Learning Goals

After this tutorial, you will gain a fundamental understanding of how to use Neurosynth Compose to conduct a meta-analysis. Specifically, you will learn about:

1. Creating a Project on Neurosynth Compose.

2. Searching for studies on PubMed.

3. Adding studies to a project.

4. Curating studies using PRISMA guidelines

5. Extracting and Annotating analyses within studies

6. Specifying a meta-analysis

7. Executing a meta-analysis

8. Displaying meta-analytic images

9. Interpreting meta-analytic results

10. Considerations for follow-up analyses.

We will be following Dr. Annie Lytical as she conducts a meta-analysis on the effects of nicotine administration on brain activity, but is in an enormous rush to get it done before the end of the day.

Story

Dear Dr. Lytical,

We hope you have settled in to your new Post Doctoral position at the University of NeuroSyncratic. It has now been a month since you started and we are excited to see what you have been working on. We would like you to send us a report of your progress so far by end of day today.

Sincerely,

Your Boss: Dr. I. M. Portant

“Oh no!, I haven’t done anything yet!” Dr. Lytical exclaimed. “I knew binge watching Bee and Puppy Cat repeatedly was a bad idea.” (And yet, she did not regret it.) She had to think of something fast. Dr. Lytical scrambled to her computer and opened up the browser. She knew some of her colleagues had been using Neurosynth Compose to conduct meta-analyses, and thought she might be able to use it to get some results, and then she could add to her meta-analysis once she had more time. She navigated to compose.neurosynth.org and was immediately struck by the beautiful design. She clicked on the “Sign In” button and was greeted with a login screen.

Fig. 1 Neurosynth Compose login screen

She signed in with her Google account and was taken to the dashboard. The most prominent element of the dashboard was Projects.

Fig. 2 Neurosynth Compose projects page

“Perfect!” she thought, I need to create a project. She clicked on the “New Project” button and a popup appeared asking for a name and description of the project. “Bugger!”, Dr. Lytical cursed under her breath, she had not decided on a topic for her meta-analysis. Quickly, she rummaged through her drawer of meta-analyses she had read recently, “Imitation is the sincerest form of flattery”, she mumbled to no one in particular as she leaved through the publications. One stuck out to her: Neurobiological impact of nicotinic acetylcholine receptor agonists. She had taken a neurobiology class in college and the class had a whole section on nicotinic acetylcholine receptors. Dr. Lytical was back in business.

Important

This tutorial is not a recommendation on how one should choose a topic for meta-analysis. One should give proper thought on the topic of their meta-analysis, and have talked to their mentors and collaborators beforehand. Please see the PRISMA statement and the 10 simple rules for neuroimaging meta-analyses. Consider registering your meta-analysis protocol on PROSPERO.

Fig. 3 Creating a new project

She typed the following information for name:

Replication of: Neurobiological impact of nicotinic acetylcholine receptor agonists

and description:

Nicotinic acetylcholine receptor (nAChR) agonists augment cognition among cigarette smokers and nonsmokers, yet the systems-level neurobiological mechanisms underlying such improvements are not fully understood. Aggregating neuroimaging results regarding nAChR agonists provides a means to identify common functional brain changes that may be related to procognitive drug effects.

After inserting the name and description, she clicked “Create” and was redirected to the following page:

Fig. 4 View of the new project.

Dr. Lytical’s next step was to search and curate the studies she wanted to include in her meta-analysis.

Searching for studies

Dr. Lytical clicked within the box “Curation: Get Started”, and was presented with three options: “PRISMA Workflow”, “Simple Workflow”, and “Custom Workflow”.

Fig. 5 Selecting the PRISMA workflow

Since Dr. Lytical is replicating a manually curated meta-analysis, she selected the “PRISMA Workflow”. PRISMA, or Preferred Reporting Items for Systematic Reviews and Meta-Analyses, is a set of guidelines for conducting systematic reviews and meta-analyses. While PRISMA encompasses the entire project workflow, Dr. Lytical has the most agency on how strictly she will abide to the guidelines during the search and curation process. Other components of PRISMA are handled by Neurosynth Compose. “Simple Workflow” is more streamlined and has one step for curation, while “Custom Workflow” allows the user to create their own steps for curation.

Fig. 6 View of the empty curation board.

She was presented with an empty curation board, with 4 sections: “Identification”, “Screening”, “Eligibility”, and “Included”.

Identification

Remove duplicate studies

Screening

Remove irrelevant studies

Eligibility

Remove studies that do not meet inclusion criteria or do not have retrievable information

Inclusion

Double check the studies and view which studies will be included in the meta-analysis

The next step is to import studies into the project.

Fig. 7 Import studies into Neurosynth Compose

After clicking on the import studies button, Dr. Lytical was presented with the following options:

Fig. 8 Options to import studies into Neurosynth Compose

Dr. Lytical has 4 options to import studies into her project.

1. Neurostore

2. PubMed

3. Manually create study

4. Citation manager export (e.g., .ris)

Neurostore

Neurostore is a database that contains all the studies that have been automatically extracted and studies that other users have shared. Every study in Neurostore is guaranteed to have coordinates or images that are amenable to meta-analysis.

PubMed

PubMed is a database of biomedical literature maintained by the National Library of Medicine. It is a traditional source for searching for studies.

Manually Created Study

If you have unpublished data or the study is not tracked by pubmed or neurostore you have the option to manually create a study.

Citation Manager Export

If you have curated a studies in your citation manager, you can get a quick start to your meta-analysis by importing the studies from your citation manager.

Dr. Lytical searches on Pubmed

Dr. Lytical decided to use PubMed to search for studies for this meta-analysis.

Fig. 9 Pubmed landing page

Hopping over to PubMed, Dr. Lytical clicked on the advanced text below the main search box, and put the following text in the advanced search box:

("fMRI" OR "PET" OR "neuroimaging" OR "Functional magnetic resonance imaging" OR "functional MRI") AND ("nicotine" OR "cigarette" OR "smok*" OR "DMXB-A") AND ("2011/01/01"[Date - Publication] : "2011/12/31"[Date - Publication])

Below is what her search looked like.

Fig. 10 Pubmed advanced search

She got the following results from her search.

Fig. 11 Pubmed search results

She downloaded the results as a PMID file, by clicking on the save button. She changed the selection to be “All Results”, and selecting the PMID format. This will download a text file with a PMID on each row.

Fig. 12 Download search results

Fig. 13 Example pmid file

We are only searching for studies in 2011 to keep the results shorter. 140 results! Oh no, that will still take too much time. Luckily Dr. Lytical remembered that she had previously narrowed down a list of PubMed ids. She found the file (linked below) and continued to the next step.

Important

TUTORIAL PMID FILE

With the pmid file in hand, Dr. Lytical went back to Neurosynth Compose and clicked on the “Import Studies” button on the top right of the curation board page.

Fig. 14 Import name

When importing studies, Dr. Lytical needs to select a search source, which could be from the Neurosynth Compose database, PubMed, a citation manager, or a manually inserted study. Since she searched on PubMed, she selected “Import Via PubMed ID List”.

Note

The Neurosynth Compose database is called “Neurostore”. It is a database of studies that have been automatically extracted from various journals, and user edited studies.

Fig. 15 Select search source

Dr. Lytical uploaded the pmid file that she had downloaded from PubMed. She could also copy and paste the pmids into the text box.

Fig. 16 Upload the tutorial pmid file

She can now view all the studies she imported as a quick visual confirmation the studies look correct.

Fig. 17 Display imported studies

Dr. Lytical named the search “manually filtered nicotinic acetylcholine receptor agonists pubmed search” and clicked “Next”. There were no duplicates, so she clicked “Complete Import”.

Fig. 18 Name the import search

Identification

All the imported studies are visible in the first column of our PRISMA board: Identification. The purpose of identification is to find duplicated studies, which is more of a concern when multiple searches are included. Since Dr. Lytical’s only source was PubMed, it is unlikely there are any duplicates. Regardless, Dr. Lytical clicked on the first study card to view the study in more detail and confirm that she can promote the study to the next Screening stage.

Fig. 19 Identification stage of curation

Fig. 20 Identification review

Dr. Lytical was now presented with the list of studies that she had imported. While the import stage has handled any obvious duplicates, the Identification stage is where she will be able to identify any duplicates that were not caught automatically. Luckily, there were no duplicates, so each study was promoted to the next stage for screening.

Screening

In this stage, Dr. Lytical is attempting to determine if the study has any relevancy to her meta-analysis. For example, the study could be about cannabis or it could be about colorectal cancer, which are topics, outside the topic of Dr. Lytical’s meta-analysis. Please read the title and abstracts along with Dr. Lytical and determine whether each study has relevancy to her meta-analysis.

Eligibility

Fig. 21 Custom exclusion

With duplicates and relevancy resolved, Dr. Lytical is now ready to determine if the study meets the eligibility criteria. In this stage Dr. Lytical likely needs to read the methods section of each study and determine if it meets the eligibility criteria. Each paper may take several minutes to evaluate, so Dr. Lytical will need to be patient and thorough. “Wait!”, Dr. Anna Lytical exclaims, “What are my eligibility criteria?”. Typically, Dr. Lytical would have to come up with the eligibility criteria on her own using her knowledge of the research question and literature to guide them. Thankfully, the eligibility criteria was already written in the protocol of the original paper, and is reproduced below.

Eligibility criteria

We included studies in this meta-analysis that: 1) employed fMRI or PET; 2) reported brain activity changes in stereotaxic coordinates (either Talairach or MNI space); 3) reported a set of coordinates (i.e., foci) from a within-subjects or between-subjects contrast assessing the effects of nAChR agonist administration (i.e., pharmacological administration or cigarette smoking) relative to a baseline condition (i.e., placebo administration or smoking-abstinence condition); and 4) examined brain activity using a cognitive or affective task paradigm or at rest (i.e., in the absence of explicit task demands). Studies examining functional connectivity, brain morphology, or neurochemistry were not included. Given the relatively modest but expanding corpus of literature regarding the impact of nAChR agonists on human brain function, no study exclusions were made on the basis of participant age, neuropsychiatric condition, or statistical threshold considerations.

Eligibility determination

While the eligibility criteria provide guidelines for inclusion and exclusion, this does not mean the answer will be obvious for each paper. Indeed, there are often disagreements between individuals reviewing the papers. In this tutorial, however, Dr. Lytical only has you, and that is enough. Please take this time to go through the papers and determine eligibility.

Inclusion/Ingestion

Hooray! Dr. Lytical has reached a critical milestone in her meta-analysis journey. The studies that survived your critical eye are happily waiting in the inclusion column. Dr. Lytical takes a quick look at the inclusion column to ensure no paper snuck through that was not supposed to survive curation. These studies are now ready to be ingested into a studyset for use in Dr. Lytical’s meta-analysis.

Fig. 22 Ingestion Button

Fig. 23 Create the studyset

Fig. 24 Create the annotation keys}

Once the studies are ingested, Dr. Lytical will be able to extract/edit coordinate data from each study and label analyses for inclusion for her particular meta-analysis. These collective labels are called annotations.

Within the context of a meta-analysis, Annotations can be explained as a way to categorize analyses within each study. For example, they can be categorized by task (e.g., Stroop, N-back, etc.), by modality (e.g., fMRI, PET, etc.), or by any other category that you want to use to filter the analyses. For this replication, Dr. Lytical is interested in the effects of nAChR agonists on the brain, which can either be excitatory or inhibitory, so she adds both an “activation” and “deactivation” column. She has can choose the data type of each column as either a Number, String, or Boolean. She selected a Boolean data type for the “activation” and “deactivation” columns, which will allow her to filter the analyses by whether they are excitatory or inhibitory. By default you will have an “include” column to help get you started.

Fig. 25 Begin Ingestion of studies into neurosynth-compose

Fig. 26 Multiple versions of the same study

The ingestion process includes looking up the study on Neurosynth Compose’s database and asking Dr. Lytical if she would want to use a pre-existing study or create a new one. If Dr. Lytical chooses to use a pre-existing study, then she will inherit all the data that was already extracted from that study, which will often include coordinates and will save her a lot of time during the extraction phase. So if there is an option to select an existing study, Dr. Lytical is highly motivated to select it. Please choose the pre-existing study on Neurostore for each study that has one.

Extraction and Annotation

Extraction and annotation are integral to a meta-analysis, and, thankfully, the Neurosynth Compose database reduces the effort for this process.

Fig. 27 View of the extraction and annotation page on Neurosynth Compose

Fig. 28 View of a particular study for annotation and extraction.

Fig. 29 View of the annotations for a particular study

Dr. Lytical is on a roll. She has now reached the extraction phase. In this phase, Dr. Lytical will extract the data from each study, specifically coordinates and annotations.

In conjunction with annotating the analyses, Dr. Lytical will also need to extract and/or edit the coordinates for each analysis. While Neurosynth Compose has many coordinates automatically extracted, Dr. Lytical will need to extract the data from some of the analyses Neurosynth Compose has not seen before. Dr. Lytical is ecstatic that so many of the studies already have coordinates extracted from the analyses.

Fig. 30 Study with no analyses

For the rest, Neurosynth Compose provides an interface to add/edit coordinates. Dr. Lytical can directly add the coordinates on the website, by copy/pasting the coordinates into the platform. It’s often easiest to copy/paste the table into google sheets first, and then select the coordinates to paste into Neurosynth Compose.

Fig. 31 Add coordinates

She skillfully add the coordinates for the remaining studies and is now ready to move on to the next stage.

Meta-analysis specification

The meta-analysis specification is the stage where many may scratch their heads and look with confusion. But not Dr. Anna Lytical, she came armed with the knowledge of learning about the different algorithms and their nuances.

Fig. 32 Meta-analysis proceed

Fig. 33 Create meta-analysis

Dr. Lytical is now ready to select the analyses and specify the algorithm. She clicks on the “Meta-Analysis Specification” button and is presented with a wizard that will guide her through the process. She names this meta-analysis: “overall effect”.

Fig. 34 Meta-analysis wizard

For this particular meta-analysis she chooses “included” and goes through a wizard to specify the algorithmic parameters for her meta-analysis.

Fig. 35 Select column included

Dr. Lytical is now ready to specify the algorithmic parameters for her meta-analysis. She chooses ALE (Activation Likelihood Estimation; Fig. 36) as the algorithm and selects 8mm as the kernel size Fig. 37.

Fig. 36 Algorithm overview

Fig. 37 Algorithm parameters

Fig. 38 Corrector parameters

Fig. 39 Parameter review

She selects the False Discovery Rate corrector (FDR) and selects “independent”, and 0.05 as the p-value threshold.

Fig. 40 Meta-analysis review

Running the meta-analysis

With the analyses selected and the algorithm chosen and specified, Dr. Lytical is now ready to run the meta-analysis. She follows the google colab link and inserts the identifier that specifies the meta-analysis. she wants to run.

Note

Fig. 41 Colab button

Fig. 42 Notebook code

Fig. 43 Insert meta analysis ID.

She executes the analysis and waits for the results. She is running a relatively small meta-analysis so she does not have to wait long for the process to finish executing.

interpreting the results

Dr. Lytical can hardly contain her excitement, but yet she must remain calm and collected, otherwise she will not be able to interpret the results correctly. It is important to remember what inferences you can and cannot make from a meta-analysis.

Looking at the top of the generated report Dr. Lytical checks to see how many studies were included in the meta-analysis and checks to see if any studies were excluded. None were excluded so she can move on to check the mask to ensure it has whole brain coverage. Seeing the mask covers the whole brain, she moves on to check the peak coordinates reported from all the analyses included in the meta-analysis. In this step, she can see which coordinates are outside the mask and if there is an unusual distribution of coordinates (e.g., are they are all on the left side of the brain? are there many coordinates in white matter tracts?).

She confirms that the meta-analytic parameters she chose in the specification match what is in the report. Then she can interactively view the results to see where the significant clusters are located. Remembering her training, Dr. Lytical recalls the Activation Likelihood Estimation creates values that can be interpreted as the probability that a voxel is activated given the studies included in the meta-analysis. The Z-values she is looking at represent the significance of that activation relative to a baseline of the voxel’s expected activation if all the foci were randomly distributed across the brain. And the Z-scores have been corrected with FDR (False Discovery Rate) correction, to help control for false positives.

She identifies clusters in the Default Mode Network (DMN) and Executive Cognition Network (ECN). In the literature, these networks are considered task negative and task positive networks, respectively. Since we did not differentiate between whether the coordinates included in the meta-analysis represented an increase or decrease in activation, Dr. Lytical cannot make any inferences about whether the nAChR agonists increased or decreased activation in these regions, merely that studies tended to report significant coordinates within these regions. As a follow up analysis, Dr. Anna Lytical could label the analyses as either “activation” or “deactivation” and run a separate meta-analysis for each category.

Conclusion

Congratulations! You’ve ran your first meta-analysis and helped Dr. Lytical keep her job.

You’ve accomplished a lot in this tutorial:

1. Created a Project on Neurosynth Compose.

2. Searched for studies about nicotine on pubmed.

3. Added studies to a project by importing them.

4. Curated studies using PRISMA guidelines within the four sections: Identification, Screening, Eligibility, and Inclusion.

5. Extracted and Annotated analyses within studies

6. Specified an ALE meta-analysis.

7. Executed a meta-analysis using google colab.

8. Displayed meta-analytic statistical images.

9. Interpreted meta-analytic results to validate and understand the meta-analysis.

10. Considerations for follow-up analyses to split up the analyses in different ways.

Dr. Lytical is planning on jumping right back in to create another Project with her own ideas, maybe something about naturalistic data while watching Bee and PuppyCat, she would gladly volunteer for that study…