The Transition From Homogeneous to Heterogeneous Machine Learning in Neuropsychiatric Research

Failure of Homogeneous and Unimodal ML in Neuropsychiatric Research

Despite recent advances in ML, a notable limitation is that brain-based predictive modeling often has low accuracy (4,15). When applying a trained model to an independent, out-of-sample population, the predicted phenotypic measure can significantly deviate from the observed value. Reviews on fMRI studies commonly report <70% accuracy in predicting diagnosis outcome of patients with various behavioral and mental health disorders (10,20). These accuracy scores, although higher than random chance, are lower than the 91% average accuracy of 503 medical imaging–based artificial intelligence (AI) models (for all types of medical diagnostics) found in a recent meta-analysis (21). In addition to the low accuracy, the compromised interpretability of these models is another critical concern (22). When interpreting a model with a 70% classification accuracy, neuroimaging biomarkers identified from the training samples do not apply to approximately 30% of the unseen population, suggesting a potential room for improvement in the generalizability and translational potential of current brain-based ML models.

There has been a rich discussion on the various reasons for low prediction accuracy (4,15), with increasing attention being paid to the issue of population heterogeneity (17,23, 24, 25). Complex demographic, socioenvironmental, physiological, and clinical factors can influence brain development, behaviors, and cognition, such that no single brain-behavior mapping likely fits all (26,27). Thus, population heterogeneity is one of the many reasons why models trained on large, consortia-sized samples (e.g., the Adolescent Brain Cognitive Development [ABCD] Study) (28) achieved notably lower accuracy than those on smaller homogeneous datasets (4,10,29, 30, 31, 32). Indeed, existing predictive analyses in neuropsychiatric research are often unimodal and homogeneous: one universal mapping function is learned to predict the target behavioral phenotype solely based on neuroimaging data and, furthermore, is expected to generalize across populations. This conceptual gap between homogeneous and heterogeneous mapping might result in the ML models being accurate only for a substratum of the population. Moreover, the interpretation of homogeneous models tends to identify neuroimaging biomarkers shared across individuals (33,34), which may as well be identified through group-level statistical tests. In fact, in many ML analyses, the prescreening or post hoc validation of neuroimaging biomarkers in turn resorts to group analysis (35,36), thus trapping what seems to be an individual-level ML analysis into a group-level analysis in disguise.

Behavioral and Neural Heterogeneity

The concept of population heterogeneity is not new. Researchers have been mostly studying the heterogeneity issue from two perspectives: precision behavioral phenotyping (Figure 1A) and neural subtyping (Figure 1B). Precision behavioral phenotyping aims to improve the understanding of behavioral heterogeneity underlying traditional clinical diagnosis: individuals meeting the criteria for a particular disorder codified by DSM exhibit significant variations in clinical symptoms, progression, and prognosis (18,24). Thus, researchers have explored alternative frameworks to harness dimensions of cognitive and behavioral constructs that complement existing diagnostic categories, enhance clinical predictions, and clarify etiology. For example, the Research Domain Criteria initiative (37) uses expert-defined dimensionally distributed quantitative traits to replace the traditional taxonomic structure of mental disorders, exploiting symptom variation across the full spectrum of severity. Beyond multidimensionality, the Hierarchical Taxonomy of Psychopathology consortium (38) identifies categorical hierarchically organized levels of diagnosis from the p factor model (39). The hierarchy encompasses both broad transdiagnostic psychopathology and narrow symptom components and thus provides a more precise and granular description of an individual’s behavioral and mental health profile. Other than factor models, various clustering and dimension reduction approaches (40, 41, 42, 43) are used for analyzing phenotypic data across multiple domains (e.g., psychopathology, personality, cognition, social functioning) to identify stable transdiagnostic clusters that cut across traditional diagnostic labels. These data-driven subtypes may exhibit greater reproducibility than those derived by hypothesis-driven methods, which could be biased by existing theories and prior assumptions.

Another type of heterogeneity analysis focuses on identifying neurobiological subtypes that support dimensions of psychopathology within a board diagnostic category. As normative modeling studies (44) of diverse brain disorders have repeatedly suggested, the specific location in the brain that indicates differences between cases and controls varies considerably across individuals with the same diagnosis (25,45). Thus, unsupervised or weakly supervised clustering is often used for revealing categorical neural subtypes (46, 47, 48), where each person is assigned to one subgroup that has more homogeneous neuropathological patterns than the overall diseased population. Alternative to categorical subtypes, neural heterogeneity can also be formulated as dimensional subtypes (e.g., by canonical correlation analysis) (49, 50, 51), where multiple constellations of neuropathological components are linked to continuous scores of specific behavioral dimensions. These biological subtypes (or subdimensions) were shown to have high correlation with within-diagnosis heterogeneity of symptom profiles (52) and have distinct treatment responses (53,54).

Relationship-Level Heterogeneity in Brain-Based Predictive ML

Despite the improved understanding of behavioral (Figure 1A) and neural (Figure 1B) heterogeneity, another type of heterogeneity that is often overlooked in building brain-based predictive ML models is the nonuniform neurobehavioral mapping (Figure 1C). In other words, the heterogeneity of interest lies not only in the neural and behavioral data themselves but also in the mapping function between them. Such mapping can reflect distinct biological mechanisms underlying the same disorder or the natural variation in brain-behavior relationships across individuals. For example, in a post hoc analysis of a single classification model that predicts cognitive test scores from fMRI data, Greene et al. (26) found that sociodemographic factors (e.g., race or level of education) are one of the main factors affecting the predictive accuracy of the model across subpopulations, an effect that was replicated in multiple datasets. In another study of 2262 children from the ABCD Study and 752 young adults from the Human Connectome Project (55), researchers show that network features predicting 33 internalizing (e.g., anxiety) and externalizing (e.g., rule breaking) behaviors were distinct across children and adults, suggesting that brain-based predictors of behaviors may change across the life span. Besides demographic constructs, environmental factors can also affect physiological functions of the brain and its ability to counteract pathological changes (56, 57, 58). A study analyzing resting-state fMRI of 6839 children from the ABCD dataset found that the correlation pattern between selective brain networks and cognitive performance varies as a function of a child’s environment (59). While better cognitive performance among children from high-income households correlated with weaker coupling between the lateral frontoparietal network and default mode network, the direction of association was reversed for children from low-income households. Taking all the evidence above, we argue that brain-based ML research may benefit from explicitly incorporating external sociodemographic and environmental factors to gain insights into multiple causal pathways from the brain to behavior.

To capture such relationship-level heterogeneity, some existing solutions rely on training, comparing, and interpreting separate models in each subcohort (27,60). For example, one can train sex-specific models (61, 62, 63) to predict disease outcome and examine whether there exist connectome patterns that are uniquely associated with males or females. While being a simple and interpretable approach, training separate models would require subcohort division based on an a priori identified categorical factor. As such, one caveat might be that model training becomes unreliable as the number of subcohorts exponentially grows with the number of considered factors and the number of samples in each subcohort drastically decreases. The separately trained ML models are more likely to overfit these smaller subcohorts (64). Alternative to defining subcohorts a priori is to first use data-driven approaches to subtype the data and then derive subtype-specific predictive models (65, 66, 67). For example, Drysdale et al. (68) first identified 4 clusters of individuals with depression symptoms and then trained separate ML models to predict treatment response to repetitive transcranial magnetic stimulation for each subtype. Not only were the subtype-specific models significantly more accurate than the universal model trained on the entire population but the neuroimaging biomarkers driving the prediction were also substantially different across subtypes. Similarly, Chen et al. (65) clustered 81 infants into 2 subgroups based on brain functional connectivity. Those subgroups had contrasting IQ in a 4-year follow-up and distinct functional connectivity patterns in neonates that predicted 4-year IQ. Although these results highlight the need to consider variability in brain-behavior relationships in brain-based ML analysis, the subtyping and predictive modeling steps are still formulated as 2 separate procedures. The successful training of the predictive models depends on the quality of derived subtypes and may fail to capture common population-level mappings.

Therefore, we believe that the community will benefit from exploring a new ML regime, which we call heterogeneous ML, to enable the discovery of interpretable relationship-level heterogeneity in an end-to-end, data-driven fashion. These models can be regarded as conditional models, wherein the relationship between input and output variables is conditioned on the moderators. Ideally, a heterogeneous ML model should be able to encode a population-level mapping function just as well as a homogeneous ML model but with the additional capacity to encode how environmental and sociodemographic factors alter that mapping. These models can not only improve understanding of brain-behavior relationship variability across individuals (69) but they also have the potential to increase the generalizability of models across populations that contain sampling bias and/or domain shifts (70,71).

Potential Approaches for Heterogeneous ML

We now turn to the question “What might be an appropriate computational approach for heterogeneous ML?” Traditional approaches for probing nonuniform brain-behavior relationships are largely based on multiple regression analysis (72), which predicts a single brain measurement from diagnosis group, behavioral phenotype, and demographic variables. Known as the moderation effects, the varying brain-behavior relationship is achieved by adding interaction terms between behavioral measurements and covariates of interest (72,73) (Figure 2A). While being simple and interpretable, these models are univariate in nature and typically analyze each voxel or region of interest independently, which can miss important dependencies among multiple brain regions (12).

To capture the multivariate brain patterns, ML predictive models reverted the direction of regression by predicting the diagnosis group or behavioral phenotype from all available brain measurements (4,10,13). However, most predictive models are homogeneous and do not consider important covariates in the training process. The challenge in identifying moderation effects in whole-brain exploratory analysis is that the high dimensional neuroimaging measurements would require a large number of interaction terms as the model input, likely resulting in training instability and overfitting. Thus, researchers need to explore alternative strategies to reliably model moderation effects in brain-based predictive models. A possible existing ML technique that could be used for this purpose is ensemble learning (74,75). For example, the mixture-of-experts algorithm aims to learn an ensemble of expert models, where each expert is specifically tailored for a subset of data (76, 77, 78). When used in brain-behavior mapping, mixture-of-experts models can define categories of relationships by learning a finite number of representative brain-behavior mapping functions that exist within the population. The final mapping for each individual is thus a weighted combination of the expert models (79) (Figure 2C) moderated by their external factors. To further interpret the encoding of moderation effects by mixture-of-experts models, future research should focus on understanding how the external factors drive the division of experts (or data subsets) and the weighting scheme for each individual.

The past few years have witnessed the increasing use of deep learning in brain-behavior mapping (10,43,54), with some preliminary evidence suggesting its superior predictive performance compared with traditional ML methods (80). These highly nonlinear models are intrinsically more expressive than traditional ML models and can be potentially better suited for learning heterogeneous mapping functions (81,82). A high-capacity deep learning model coupled with large-scale data, in principle, can implicitly extract the information related to latent factors from the neuroimaging data (83, 84, 85) and subsequently use that information to learn nonuniform brain-behavior mapping within the population. To explicitly link those latent factors with interpretable environmental and sociodemographic factors, multimodal predictive models can be used to combine imaging and nonimaging (e.g., tabular) data as the input (i.e., multimodal data fusion) (86, 87, 88, 89) to learn factor-modulated mapping functions. However, identifying whether and how external factors interact with neuroimaging measures or whether they are simply learned as final additive effects can be challenging. One potential way to increase the interpretability of relationship-level heterogeneity in deep learning models is to use a hypernetwork (90) mimicking the interaction term in traditional regression analysis. A hypernetwork is a neural network that generates a subset of parameters in a deep learning model. As opposed to the previous categorical setup in mixture-of-experts, this type of relationship-level heterogeneity captures dimensional moderation effect, as the parameters of the core brain-behavior mapping network are continuously moderated by external factors represented within the second hypernetwork (Figure 2B). Given that deep networks are characterized by hierarchical extraction and abstraction of features from input data (91), one has the flexibility to choose which specific layer(s) of the network is moderated, thereby bypassing the need for including the full interaction of external variables with whole-brain connectome measurements.

Upcoming Challenges and Next Steps

Despite the vision that heterogeneous ML might help unlock the mystery of nonuniform mapping between biology and psychopathology, some key components require careful consideration.

Conclusions

Many challenges need to be addressed before brain-based modeling can substantially advance our understanding of complex brain-behavior relationships and subsequently translate to improved health care. Here, we reviewed current research analyzing population heterogeneity and identified one underexplored problem: the current design of brain-based ML models fails to examine the heterogeneous nature of brain-behavior mappings. Given that converging evidence points to the benefit of modeling relationship-level heterogeneity, we acknowledge that a more comprehensive study design is required to test the efficacy of heterogeneous ML and evaluate its improvement over unimodal and homogeneous ML. In pursuing this path, understanding how to accurately model moderation and confounding effects of environmental and sociodemographic factors, along with optimally evaluating and interpreting factor-moderated ML models, remain key challenges. Once addressed, ML can unleash its unique power to quantify and disentangle brain-behavior-factor relationships to truly personalize medicine.