Leveraging the adolescent brain cognitive development study to improve behavioral prediction from neuroimaging in smaller replication samples

Introduction

Understanding how the brain gives rise to behavior is a central goal and challenge of modern neuroscience. Non-invasive neuroimaging techniques have yielded valuable opportunities to link brain structure and function with cognitive and mental health phenotypes, which in turn could be useful to predict later behavioral outcomes (Kardan et al. 2022). However, recent reports have provided evidence that call into question the reproducibility of brain-behavior associations across various magnetic resonance imaging (MRI) modalities (Elliott et al. 2020; Kelly Jr and Hoptman 2022; Kennedy et al. 2022; Marek et al. 2022; Liu et al. 2023), even in large samples such as the Adolescent Brain Cognitive Development^SM Study (ABCD Study®), which have yielded much smaller effect sizes than anticipated (Dick et al. 2021).

Recent reports have suggested that studies require thousands of individuals to be sufficiently powered to measure reproducible brain-behavior associations when applying commonly used univariate statistical methods (Marek et al. 2022; Liu et al. 2023). This has resulted in significant concerns across the brain imaging community, and beyond, about the value of MRIin trying to understand human behavior, particularly for the preponderance of investigations that have been conducted in sample sizes well under one hundred, let alone thousands, of individuals. Dozens of research groups and media outlets have independently responded to this highly cited claim of thousands of individuals being required in brain-wide association studies, including many commentaries (“Cognitive neuroscience at the crossroads” 2022, “Revisiting doubt in neuroimaging research” 2022; Bandettini et al. 2022; Deyoung et al. 2022; Gratton et al. 2022; Kong et al. 2022; Rosenberg and Finn 2022; Tiego and Fornito 2022; Valk and Hettwer 2022; Chakravarty 2022; Uddin 2023), reports using synthetic data (Cecchetti and Handjaras 2022; Gell et al. 2023), neuroimaging/cognitive data from the UK Biobank and/or Human Connectome Project (Cecchetti and Handjaras 2022; Gell et al. 2023; Spisak et al. 2023), and psychopathology/genetic data from the ABCD Study (Tiego et al. 2023). Across this wave of responses, it has become increasingly clear that hundreds of individuals should be sufficient to measure reproducible brain-behavior associations, particularly with the right choice of multivariate prediction methods (Spisak et al. 2023) and a more intentional focus on specific behavioral phenotypes.

It remains to be seen how such multivariate methods enhance power to predict behavior in a neurodevelopmental sample such as the ABCD Study. Further, many reports focusing on brain-behavior reproducibility issues have focused on resting state functional MRI (fMRI), necessitating a broader investigation of different imaging modalities, including structural and diffusion MRI, and task-evoked brain activation with fMRI. Marek, Tervo-Clemmens and colleagues (Marek et al. 2022) examined both multivariate methods and task fMRI, but did not highlight these results in the main conclusions of their paper, and used a multivariate technique that may not have been optimal in predicting cognition (Spisak et al. 2023). Task fMRI is of particular interest in the prediction of cognitive performance, given its success in mapping patterns of brain activity evoked by different tasks, identifying both distributed and specific neural responses to different processing demands. Our group and others have recently shown that task-evoked activation outperforms resting state functional connectivity measures when predicting cognition in the ABCD study (Chen et al. 2022; Omidvarnia et al. 2023; Zhao et al. 2023), and similar conclusions have been drawn in other samples of youth and adults (Rosenberg et al. 2016; Greene et al. 2018; Jiang et al. 2020; Finn and Bandettini 2021).

Here we compare the sample sizes required to detect reproducible brain-behavior associations across imaging modalities using univariate and multivariate methods. We demonstrate that by applying multivariate prediction methods that are sensitive to the complex familial structure of the ABCD study, we can capture significant patterns of associations between behavior and lower dimensional representations of structural/functional brain architecture, reducing the burden of multiple comparisons common to univariate studies. Multivariate analyses in turn yield larger effect sizes and more reproducible patterns of brain-behavior associations, negating the need for thousands of individuals, and instead showing sufficient power can be reached with well under one hundred individuals.

Materials and methods

Results

Discussion

Our work demonstrates that by leveraging large datasets such as ABCD, reproducible brain-behavior associations can be measured with multivariate methods applied to structural and functional MRI in smaller replication samples (e.g. approximately 100 subjects) typical of many existing publications, grants, and databases. We find that a working-memory functional MRI task is particularly well-powered to predict general cognition in the baseline ABCD sample, with only ~ 41 subjects required to achieve 80% power, and this is further boosted to just 29 subjects when predicting a measure of performance on the fMRI task itself. Finally, we show that with 100 random samplings of just 100 subjects in the discovery sample, prediction can be adequately powered with just 66 subjects in the replication sample when using multivariate methods to predict cognition from working memory task-fMRI data. This work paints a much more optimistic landscape of opportunities offered by non-invasive MRI techniques across different imaging modalities with even several dozen subjects, particularly with targeted experimental designs and improved statistical analysis.

The boost in power afforded by multivariate as compared to univariate methods in measuring brain-behavior associations is well-documented (Sripada et al. 2020; Palmer et al. 2021; Zhao et al. 2021; Marek et al. 2022; Spisak et al. 2023). Associations between cognitive performance and neuroimaging phenotypes are particularly amenable to multivariate methods, given the distributed and sparse nature of these effects across the brain (Palmer et al. 2021; Zhao et al. 2021). Functional MRI analysis has long capitalized on observed strong patterns of spatial and temporal correlations across the brain (Derado et al. 2010). Structural imaging-derived phenotypes also have a strong covariance structure between regions (Lerch et al. 2006), likely due to gradients of core neurodevelopmental genetic factors shaping the cortex early in life (Rakic 1988, 2009; van der Meer et al. 2020; Makowski et al. 2022). In line with our previous work (Palmer et al. 2021), multivariate methods capturing distributed brain structural patterns explained a larger amount of variance in cognition compared to univariate mapping. We observed weak univariate effect sizes between structural/diffusion imaging measures and cognitive measures, with the lowest effects for intracortical restricted diffusion. However, and importantly, these small effects with sMRI and RS fMRI received substantial boosts with multivariate modeling. Our results emphasize the importance of taking into consideration structural and functional brain patterns as a whole in brain-behavior analyses, rather than analyzing any single vertex or region-of-interest in isolation. Although results were weaker overall with measures of psychopathology (e.g. internalizing and externalizing symptoms, p factor of psychopathology), these measures still benefited from a boost in prediction, particularly when predicted by fMRI and surface area, using multivariate methods. Generally, we also observed that surface area and task fMRI performed quite well for other cognitive measures, once again suggesting that replication sample sizes still only require on the order of hundreds, not thousands, of individuals for these imaging measures.

The notably high prediction performance of the ENb task on general cognition and accuracy on 2-back trials showcases the power of using a task-evoked functional MRI study design in predicting task-relevant behavior. Our group has previously shown that accuracy on 2-back trials was highly correlated with total composite cognition (Zhao et al. 2023), while performance on a motor inhibitory task exhibited little or no correlation with either cognitive measure. This behavioral correlation pattern was consistent with the greater prediction performance of the ENb task on total composite cognition, and the weak associations found with the motor inhibitory task. We also demonstrate that the choice of task fMRI contrast is an important variable in a predictive framework, whereby we only saw a boost in predictive performance with the 2- vs 0-back contrast, and not for other contrasts that do not engage working memory-related processes to the same extent. Although structural MRI and resting-state fMRI can be useful in mapping brain-behavior associations, our results do emphasize the advantage of using an active experimental functional brain imaging paradigm to improve predictive performance of behavior.

Similar to the structural MRI measures, we find that resting state fMRI predictors also receive a generous boost from multivariate prediction methods (r = 0.286 and 94 subjects to detect with 80% power for general cognition) compared to univariate (r = 0.116 and 584 subjects needed). Beyond general cognition, the resting state correlation matrix predicted other cognitive and psychopathology measures in a manner similar to cortical thickness predictors. However, our ENb task fMRI predictors still outperformed resting state fMRI across all behavioral outcomes. There was also a large degree of variability in resting state fMRI prediction across iterations, as can be seen by the larger adjusted standard deviation estimates, particularly for multivariate prediction. We also further emphasize in this work, concordant with recent work from our group (Zhao et al. 2023), that task fMRI outperforms resting state fMRI in predicting task-relevant behavior (i.e. accuracy on 2-back trials). Altogether, this suggests that low-dimensional patterns of resting-state correlations across the brain are more useful than any single pair-wise correlation on its own in predicting behavior. However, the absence of an actively engaging task with resting state fMRI may be contributing to much more heterogeneous performance metrics across different subsamples, compared with the more stable estimates derived from task fMRI. Finally, we found very poor performance of functional correlations between pre-defined networks that are frequently incorporated into neuroimaging analyses, calling into question the validity and biological interpretation of these networks when defined in independent samples.

We did not observe large boosts in effect sizes with diffusion MRI measures when comparing multivariate to univariate associations. RND within superficial white matter showed a slight boost, with 290 participants required in the replication sample to achieve 80% power in predicting cognition with multivariate methods, compared with 508 with univariate methods at the same level of power. Notably, RND, derived from restriction spectrum imaging modeling, benefited more from multivariate methods for prediction of general cognition compared to fractional anisotropy of superficial white matter. The slight improvement in performance of RND compared to FA is consistent with several reports that RSI measures, derived from multi-shell diffusion imaging acquisitions, may be more sensitive in capturing microstructural properties especially in the context of neurodevelopment (Palmer et al. 2022), and neurological and psychiatric disorders (Carper et al. 2016; Loi et al. 2016; Reas et al. 2018). For restricted isotropic diffusion intracortically, both methods still required over 1000 individuals. Although the current manuscript focused on peri-cortical associations (i.e. in and around the cortex), it cannot be ruled out that diffusion measures of other brain regions, that may be more reliably measured with MRI (e.g. deeper white matter tracts, subcortical structures), would still benefit from multivariate analyses. Future work extending multivariate analyses to voxel-wise prediction of cognitive and other behavioral phenotypes hypothesized to be more strongly associated with microstructure would be fruitful in testing this hypothesis.

The conclusion reaching many headlines that thousands of individuals are required to measure brain-behavior associations with structural and functional MRI (Marek et al. 2022) is based on the assumption that researchers are working within a univariate framework and sampling across a broad array of outcomes. This conclusion was also based largely on the poor performance of resting-state functional connectivity to predict cognition and other behaviors, a method that does not orient participants’ attention to any particular task or stimulus, and in turn, may not be the best candidate for predicting measures of cognition (Rosenberg and Finn 2022; Zhao et al. 2023). This “brain-wide association study” approach of sampling across a large array of behavioral outcomes with mass univariate analyses bears some resemblance to methods used in genome-wide association studies (GWAS). However, the application of multivariate methods helps ease the burden of multiple comparison corrections that mass univariate methods typically require. Instead, our findings highlight a different use case of a GWAS-like framework, where we can leverage a larger discovery sample, such as the one afforded by the ABCD Study, to define multivariate brain patterns associated with behavioral measures of interest, and in turn, inform findings in smaller samples. In this vein, our approach parallels that of the application of consortia-led GWAS to the derivation of polygenic risk scores in independent datasets, where large samples can help us obtain more precise predictive weights for out-of-sample prediction.

However, large “ABCD-like” samples are not always necessary if working with strong brain-behavior relationships to begin with. Here we highlight analyses testing out-of-sample prediction performance given different discovery sample sizes, showing that even with 100 random samplings of 50 subjects in the discovery sample, only ~ 100 subjects are required in replication to predict general cognition from task fMRI data using multivariate methods. It is important to note, however, that there is still large variability in prediction estimates across 100 iterations at smaller discovery samples. Additionally, integration of regularization methods in our multivariate prediction framework may have contributed to the well-powered results even in smaller discovery samples by reducing the chances of overfitting, leading to a more optimal model. This further emphasizes the fact that with the correct methodological choices, thousands of individuals are not always required in either discovery or replication samples in pursuit of meaningful and reproducible brain-behavior associations. This is particularly evident when predicting relevant in-scanner behavior (i.e. 2-back accuracy) from task activation during the ENb task, where only 29 replication subjects were required to assess this relationship with 80% power. Larger samples, however, are required for other weaker brain-behavior associations, for instance with internalizing and externalizing measures of psychopathology in ABCD.

We have previously shown in our own work that a notable portion of the variance attributed to imaging measures in predicting cognition is shared with sociodemographic variables (Zhao et al. 2023), reflecting the complex interplay between socioeconomic resources and other environmental factors, brain development and behavior. Others have also shown the importance of socioeconomic resources on brain-behavior relationships in models integrating both structural (Thomas and Coecke 2023; Farah 2017; Brito and Noble 2014; Raizada and Kishiyama 2010) and functional (Tomalski et al. 2013; Tomasi and Volkow 2023; Demir et al. 2015) imaging modalities. Sociodemographic factors capture a complex array of variables that differentially and jointly influence brain structure. For instance, in a large developmental sample of youth between the ages of 3 and 20 years, parental education and family income were non-linearly associated with cortical surface area, with most pronounced effects at lower income levels and in heteromodal brain regions, such as frontal and temporal areas important for language and memory (Noble et al. 2015). Socioeconomic resources are also hypothesized to modulate neurodevelopmental trajectories (Thomas and Coecke 2023), where evidence has suggested altered development of gray matter (Hanson et al. 2013), cortical thickness (Piccolo et al. 2016) and myelin-based markers (Ziegler et al. 2020) with age as a function of socioeconomic factors. These neuroanatomical and functional relationships with variables such as poverty levels, household income, and parental education may be an integrated result of the impact of socioeconomic resources on cognition in youth, reflected through both neural and behavioral mechanisms (Ursache and Noble 2016). We acknowledge that socioeconomic status is a highly complex, multi-dimensional construct that would require the inclusion of a larger array of environmental, community-based and developmental measurements to better understand its influence on cognition, which could form the basis for future work using the ABCD Study dataset.

The included age range of participants in this work (ages 9–11 years) marks a time of dynamic development with respect to both neurocognition and brain maturation, captured in part by our MRI-based cortical measures. Alterations in synaptic density within the cortical mantle are intricately linked with cognitive development (Rakic et al. 1994; Petanjek et al. 2023; Petanjek et al. 2011). Sex differences in cortical trajectories with age have also been shown putatively linked to differences in synaptic and dendritic architecture (Duerden et al. 2019). Although we only focused on a single timepoint and pooled across the sexes to remain consistent with other recent work in this domain, it will be important to integrate incoming longitudinal data from the ABCD Study, to elucidate if any imaging modalities or brain patterns may have more predictive power of cognition in a neurodevelopmental context. Sex-dependent maturational patterns may also yield important clues for cognitive performance and neuropsychiatric measures that show sex differentiation.

We recognize that although effects were strongest with task fMRI compared to other imaging modalities, the included sample of 5,673 individuals may not generalize to the full sociodemographic spread of the ABCD baseline sample. Similarly, our sample with recommended resting state fMRI data for inclusion (n = 5,321) represented the smallest sample of all of the imaging modalities. We aimed to stay as consistent as possible with other recent work testing the reproducibility and power of brain-behavior associations with neuroimaging, and restricted our covariates to age, sex, and scanner type. However, future investigations would benefit from a deeper understanding of the impact of sociodemographic and environmental factors in behavioral prediction paradigms, as discussed above. The brain-behavior associations reported here may not generalize to older adolescents or adults, given the age range of the included ABCD baseline sample. Future work is encouraged to integrate longitudinal data from ABCD to boost effect sizes of brain-behavior associations (Kang et al. 2023), as well as integrate independent datasets from different developmental stages for broader generalization and external validation. Given the relatively weaker predictive power of resting-state fMRI compared to task fMRI for cognition shown both in this work and others (Zhao et al. 2021; Marek et al. 2022), there has been increased dialogue surrounding a potential paradigm shift from resting state to task-based functional MRI measures (Greene et al. 2018). Mental health-related phenotypes are of particular interest in adolescent samples, but show weaker effects with brain structure and function in ABCD Study data compared to general cognition. Future directions would benefit from a focus on modeling of behavioral phenotypes to improve precision behavioral phenotyping (Tiego et al. 2023). It will also be important to integrate independent datasets for external validation. Finally, recommendations have been made to steer away from single modeling approaches and understand how various models (e.g. mixture modeling) (Greene et al. 2022) could better capture the complex relationships between brain and behavior reflected in large diverse datasets such as ABCD.

We show that reproducible brain-behavior associations can be obtained with dozens, rather than thousands, of individuals, helping to quell growing concerns of the demise of reproducible results with neuroimaging. Our findings help shed clarity on the utility of neuroimaging studies, particularly for understanding normative and aberrant neurodevelopment, and present a more hopeful view for future funding priorities of smaller neuroimaging studies and policy decisions.

Supplementary Material

Author contributions

Carolina Makowski (Conceptualization, Data curation, Formal analysis, Investigation, Project administration, Validation, Visualization, Writing—original draft, Writing—review & editing), Timothy T. Brown (Conceptualization, Writing—review & editing), Weiqi Zhao (Data curation, Methodology, Writing—review & editing), Donald J. Hagler (Data curation, Software, Writing—review & editing), Pravesh Parekh (Methodology, Writing—review & editing), Hugh Garavan (Writing—review & editing), Thomas Nichols (Methodology, Writing—review & editing), Terry Jernigan (Conceptualization, Investigation, Project administration, Writing—review & editing), Anders Dale (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing—review & editing).

Funding

Data used in the preparation of this article were obtained from the Adolescent Brain Cognitive DevelopmentSM (ABCD) Study (https://abcdstudy.org), held in the NIMH Data Archive (NDA). This is a multisite, longitudinal study designed to recruit more than 10,000 children age 9–10 and follow them over 10 years into early adulthood. The ABCD Study is supported by the National Institutes of Health (NIH) and additional federal partners under award numbers U01DA041048, U01DA050989, U01DA051016, U01DA041022, U01DA051018, U01DA051037, U01DA050987, U01DA041174, U01DA041106, U01DA041117, U01DA041028, U01DA041134, U01DA050988, U01DA051039, U01DA041156, U01DA041025, U01DA041120, U01DA051038, U01DA041148, U01DA041093, U01DA041089, U24DA041123, and U24DA041147. A full list of supporters is available at https://abcdstudy.org/federal-partners.html. A listing of participating sites and a complete listing of the study investigators can be found at https://abcdstudy.org/study- sites/. ABCD consortium investigators designed and implemented the study and/or provided data but did not necessarily all participate in analysis or writing of this report. This manuscript reflects the views of the authors and may not reflect the opinions or views of the NIH or ABCD consortium investigators. The ABCD data repository grows and changes over time. Data were drawn from the NIMH Data Archive ABCD Collection Release 4.0 (DOI: 10.15154/1523041).

Additionally, this work was supported by the National Institutes of Mental Health, award number (K99MH132886 to C.M.), the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement (number 801133 to P.P.), and the Research Council of Norway (grant number 324252 to P.P.).

Conflict of interest statement: Dr Dale reports that he was a Founder of and holds equity in CorTechs Labs, Inc., and serves on its Scientific Advisory Board. He is a member of the Scientific Advisory Board of Human Longevity, Inc. He receives funding through research grants from GE Healthcare to UCSD. The terms of these arrangements have been reviewed by and approved by UCSD in accordance with its conflict of interest policies. Dr Dale also reports that he has memberships with the following research consortia: Alzheimer’s Disease Genetics Consortium (ADGC); Enhancing Neuro Imaging Genetics Through Meta Analysis (ENIGMA); Prostate Cancer Association Group to Investigate Cancer Associated Alterations in the Genome (PRACTICAL); Psychiatric Genomics Consortium (PGC). All other authors have no conflicts of interest.