Figure 2.
A snapshot of translational neuroimaging using multivariate predictive models. We searched PubMed for original neuroimaging research articles (including EEG, positron-emission topography (PET), MRI, diffusion tensor imaging (DTI) and arterial spin labeling (ASL)) published between 1983 and January 2016. The search terms can be found via this link: http://goo.gl/N7oh0i. Nonhuman and nonclinical studies were excluded, as well as those that did not employ multivariate pattern recognition. The initial search yielded 2,767 studies, of which 536 studies were selected based on review of their abstracts. Full-text review was used to select 475 studies that included 615 classification or predictive maps. (a) Top: growth of pattern recognition studies in translational neuroimaging since 2004. Bottom: growth of sample sizes in translational neuroimaging studies. The y-axis shows the largest sample size among studies published each year. (b) Breakdown of studies by diagnostic category. PTSD, post-traumatic stress disorder. ADHD, attention deficit hyperactivity disorder. (c) Uses of pattern recognition models. ‘Diagnosis’ refers to patient vs. control classification and ‘risk group’ to classification of groups at high risk (for example, relatives of people with disorders) vs. controls. ‘Symptom’ refers to prediction of continuous symptom scores. ‘Subtype’ refers to identification of subgroups of patients based on brain patterns. ‘Prognosis’ and ‘treatment response’ refer to predictions of individual differences in disease progression and response to an intervention, respectively. ‘Component process’ studies identify predictive models for basic cognitive or affective processes and apply those to classifying patient groups or to predicting symptoms in patients. (d) Precision-weighted accuracy, based on the square root of the sample size, for patient vs. control classification in model-development samples. Here we show classification accuracy only for patient vs. control classification, which was the most common use across disorders (75% of predictive models). The size of the circles shows the precision estimates, with larger circles indicating larger samples and more precise estimates. Accuracy was nearly always estimated using cross-validation. (e) Classification results from prospective testing on independent data sets. Only a small minority of studies report prospective tests. Lower accuracy in independent tests is indicative of bias in cross-validated accuracy estimates from training samples. Accuracy is lower in most cases reviewed here, with AD classification showing least evidence for bias. (f) Diagnostic classification accuracy as a function of sample size for six types of disorders. As the estimates from the largest studies are the most precise, they are most representative of the true accuracy. Across disorders, very high classification accuracy is reported in some small studies, but these have not been replicated in prospective tests. With a few exceptions, accuracy values for large-sample studies are much more modest. These observations point to the need for improvements in statistical model development, data aggregation and prospective testing of promising models across multiple, diverse samples.