. Author manuscript; available in PMC: 2026 Feb 4.

Published in final edited form as: IEEE Trans Med Imaging. 2025 Feb 4;44(2):941–951. doi: 10.1109/TMI.2024.3467384

An Explainable Unified Framework of Spatio-Temporal Coupling Learning with Application to Dynamic Brain Functional Connectivity Analysis

Bin Gao ¹, Aiju Yu ², Chen Qiao ³, Vince D Calhoun ⁴, Julia M Stephen ⁵, Tony W Wilson ⁶, Yu-Ping Wang ⁷

PMCID: PMC11977455 NIHMSID: NIHMS2054419 PMID: 39320999

Abstract

Time-series data such as fMRI and MEG carry a wealth of inherent spatio-temporal coupling relationship, and their modeling via deep learning is essential for uncovering biological mechanisms. However, current machine learning models for mining spatio-temporal information usually overlook this intrinsic coupling association, in addition to poor explainability. In this paper, we present an explainable learning framework for spatio-temporal coupling. Specifically, this framework constructs a deep learning network based on spatio-temporal correlation, which can well integrate the time-varying coupled relationships between node representation and inter-node connectivity. Furthermore, it explores spatio-temporal evolution at each time step, providing a better explainability of the analysis results. Finally, we apply the proposed framework to brain dynamic functional connectivity (dFC) analysis. Experimental results demonstrate that it can effectively capture the variations in dFC during brain development and the evolution of spatio-temporal information at the resting state. Two distinct developmental functional connectivity (FC) patterns are identified. Specifically, the connectivity among regions related to emotional regulation decreases, while the connectivity associated with cognitive activities increases. In addition, children and young adults display notable cyclic fluctuations in resting-state brain dFC.

Keywords: Spatio-temporal coupling, explainability, spatio-temporal evolution mechanism, brain network analysis, dynamic functional connectivity

I. Introduction

WITH the availability of spatio-temporal data, it becomes increasingly important to explore the coupling relationships between spatio-temporal information, which can provide insights into the underlying biological mechanisms. This is particularly important for functional magnetic resonance imaging (fMRI), where spatio-temporal information is available for brain functional network analysis. In the chronnectome, coupling refers to time-varying correlations or interactions in activity levels between brain regions, whose spatial characteristics may also evolve [1]. Spatio-temporal data typically present temporal and/or spatial dependencies [2]. However, existing learning techniques often model temporal and spatial information independently without considering their potential coupling relationships, resulting in an incomplete picture of the underlying relationships. Integrating temporal and spatial information is thus the key.

Deep learning-based methods are increasingly being applied to model the spatio-temporal patterns of functional brain networks. For instance, the convolutional auto-encoder (CAE) has been utilized to model the temporal patterns of functional brain networks, followed by a regression model to obtain the spatial patterns [3]. The restricted Boltzmann machine has been applied to model the temporal patterns of brain functional networks, while the spatial patterns can be obtained via LASSO [4]. Zhao et al. proposed a model called spatio-temporal convolutional neural network to extract spatial and temporal characteristics from targeted networks jointly [5]. Xiao et al. developed a dynamic decomposition model to explore the spatio-temporal structure of brain dynamics. Based on this model, dFC is decomposed into a linear combination of dynamic modules and time-varying weights. Moreover, the variability of the time-varying weights is conceptualized as the flexibility of the corresponding dynamic modules [6]. Graph neural networks (GNNs) are also applied to extract spatio-temporal information. For instance, Yan et al. proposed a novel multi-head guided attention graph neural networks model, which can simultaneously model the spatio-temporal patterns of the individual brain’s overall functional networks and further predict individual behaviors [7]. El-Gazzar et al. El-Gazzar et al. proposed a dynamic adaptive spatio-temporal graph convolution (DAST-GCN) model to overcome the shortcomings of pre-defined static correlation-based graph structures. The proposed approach uses dilated 1-D convolution for temporal feature extraction and allows end-to-end inference of dynamic connections between brain regions via a layerwise graph structure learning module [8]. Ye et al. proposed a biologically-inspired architecture, called as spatio-temporal pyramid graph convolutional network (STpGCN), to capture spatio-temporal graph representation of brain activities. By designing multi-scale spatio-temporal pathways and bottom-up pathways that mimic the information process and temporal integration in the brain, the STpGCN can achieve high brain decoding performance [9]. Zhu et al. used the optimal transport theory to comprehend the topology evolution of dynamic brain networks. They devised a multi-channel spatio-temporal graph convolutional network (MCSTGCN) with an attention mechanism. This model collaboratively extracts spatial and temporal features from the evolution networks [10]. Azevedo et al. proposed a deep neural network architecture. It combines graph neural networks and temporal convolutional networks, which can learn from the spatial and temporal components of rs-fMRI data in an end-to-end fashion [11]. Huang et al. introduced a spatio-temporal directed acyclic graph with attention mechanisms with different sizes of convolutional kernels to learn multiscale features from spatio-temporal graphs [12]. Furthermore, considering the rich temporal information in brain dFC, recurrent neural networks (RNNs) have been used for modeling temporal dynamics. For instance, Kong et al. proposed a novel spatio-temporal graph convolutional network (STGCN) framework, constructing the spatial graph attention convolution module to enhance the spatial-learning capability of sliding windows while utilizing long short-term memory to learn the differences between sliding windows [13]. Yin et al. applied multiple GCNs to extract graph domain features and used long short-term memory to learn the temporal changes between two channels within a specific time window and extract temporal features [14]. Lian et al. proposed a pedestrian trajectory prediction model based on a spatio-temporal graph convolutional network (PTP-STGCN) which combines the graph convolution network, temporal transformer network, and prediction layer based on the time-extrapolator convolutional neural network to capture the interaction between graph nodes in the spatial and temporal dimensions [15]. Jiang et al. developed an anatomy-guided spatio-temporal graph convolutional networks (AG-STGCN) model, which utilizes spatio-temporal characteristics of 4-D fMRI to investigate the gyro-sulcal FC characteristics within a single and across different task domains [16].

Despite the success of the above deep learning-based models, there are still several challenges:

1). Latent Coupling Relationships within Spatio-Temporal Data:

Existing methods often independently model temporal and spatial information, failing to capture rich spatio-temporal coupling relationships, let alone construct a unified learning framework for spatio-temporal data.

2). The Explainability of the Model:

For explainability, it is defined as the ability to specify the data features which contribute the most to decision-making [17]–[19]. It can help us better understand rationales behind the model decisions, build trust, and potentially provide feedback for further improvement [18]–[21]. Currently, research on the explainability of spatio-temporal learning models is limited and often neglects the coupling relationships between spatial and temporal data, which can not completely describe the dynamic interaction of spatio-temporal data, let alone understand their underlying evolution mechanism.

3). Brain dFC Analysis:

Previous studies have shown that the FC of brain networks varies over time and is associated with cognitive activities [22]. However, current models often focus on extracting feature information using a priori graph structures and overlook the change of brain FC matrices over time [23], [24].

This paper will address the above issue with the following approaches.

1). Constructing a Unified Framework of Spatio-Temporal Coupling Learning:

The proposed framework consists of three interrelated modules designed to simultaneously consider temporal and spatial correlations across both time and space. This framework provides a unified model for analyzing spatio-temporal data, enabling the exploration of their coupling relationships for improved prediction.

2). Building an Explainable Learning Model:

The explainability of a model is demonstrated by its capability of comprehending the data through revealing the causal relationships between the features and the predicted outcome. For a specific task, at each time step, the node representation vectors and connection representation matrices are updated through alternative iterations. The learned node vectors and connection matrices can therefore show how the spatio-temporal coupling relationships evolve, providing the explainability.

3). Validating and Applying the framework to Brain dFC Analysis:

The proposed unified framework is applied to brain dFC analysis. Significant developmental differences in dFC and changes in brain evolution patterns during resting state can be revealed. Compared to the existing methods, the spatio-temporal coupling framework can simulate dynamic mechanisms that are in line with the working mechanisms of human brain, providing an explanation of the inherent changes in brain development. The model can be applicable to analyze spatio-temporal data in other fields.

In summary, this paper proposes an explainable spatio-temporal coupling unified learning framework with applications to brain dFC analysis based on the resting-state fMRI data from the Philadelphia Neurodevelopmental Cohort. The goal is to explore the differential brain dFC during the developmental process and the spatio-temporal evolution mechanism of the brain in the resting state.

II. STCUF-EX and its applications to the Analysis of Brain dFC

A. STCUF-EX

The proposed spatio-temporal coupling unified framework with explainability (STCUF-EX) is presented as follows:

{\begin{matrix} {\tilde{H}}_{t} = T A (N R ({\tilde{A}}_{t - 1}, H_{t}), {\tilde{H}}_{t - 1}) \\ {\tilde{A}}_{t} = T A (C R (A_{t}, {\tilde{H}}_{t}), {\tilde{A}}_{t - 1})) \end{matrix}

(1)

It consists of three main modules: the temporal analysis $(T A)$ module, the node representation $(N R)$ module, and the connection representation $(C R)$ module. $H_{t}$ represents the node representation vector at time $t$ , and $A_{t}$ represents the connection representation matrix at time $t$ . ${\tilde{H}}_{t - 1}$ denotes the node representation vector obtained from the iterative output at time $t - 1$ , while ${\tilde{A}}_{t - 1}$ represents the connection representation matrix obtained from the iterative output at time $t - 1$ .

The equation shows that the designed spatio-temporal coupling is based on two components: the connection representation matrix and the node representation vector, which are interdependent on each other during the update process. The learned node representation vector is used to update the connection representation matrix, which feeds back into the learning in the next iteration. The same update applies to the connection representation matrix. By updating alternately over time, the coupling can be achieved. The details of each module are described as follows:

1). Timing Analysis ( $TA$ ):

$T A$ model is designed to iteratively update the node representation and the connection representation in spatio-temporal coupling learning, and it operates across the entire training process. As shown in Fig. 1, each $T A_{t}$ represents a module of the model at time point $t$ .

Fig. 1. — The flowchart of STCUF-EX and its application in brain dFC analysis. The left subfigure shows the data preprocessing procedure, encompassing the re-partitioning and representation of the raw rs-fMRI data using an ROI template. Subsequently, the sliding window technique obtains node and connection representations for each sub-brain at each time step. These outcomes serve as the input data for the model. For the middle one, the specific model framework of STCUF-EX is illustrated for dynamic brain functional connectivity analysis. Here, NR represents the node representation module, CR represents the connection representation module, and $T A_{t}$ represents a module of the timing analysis model at time point $t$ , which operates across the entire training process and resolves long-term dependency issue. The mutual interaction relationships among the modules are illustrated by Eq. (1). Concretely, at each time step, the model receives the node representations and connection representations from the previous step, combines them with the current sub-brain node and connection information, and jointly learns updated node and connection representations for subsequent time step. The right subfigure is the classification decision based on the learned node representations, and the feature selection using dual-sample $t$ -test based on the learned functional connectivity matrix.

Specially, it utilizes the learned history ${\tilde{H}}_{t - 1}$ , and the node vector learned by the $N R$ module, to update the node representation ${\tilde{H}}_{t}$ . The $N R$ module combines the initial input of the node vector $H_{t}$ and the learned connection ${\tilde{A}}_{t - 1}$ at time point $t - 1$ to achieve the node vector at the current time $t$ , coupling spatial and temporal information. The connection ${\tilde{A}}_{t}$ is updated by $T A_{t}$ in a similar manner. In this way, the information across the entire training process, i.e., ${\tilde{H}}_{t}$ and ${\tilde{A}}_{t}$ , can be retained for subsequent learning with improved performance due to spatio-temporal coupling. As a result, $T A$ model can help resolve the long-term data dependency issue.

$T A$ provides a general approach for long-term dependency learning in time series, which includes long short-term memory (LSTM), and gate recurrent unit (GRU). In this paper, we design the $T A$ model based on GRU to capture dynamic connectivity changes.

2). Node Representation Module (NR):

The function of $N R$ is to update the node representation vector at the current time step based on the previously learned connection representation matrix as follows:

N R ({\tilde{A}}_{t - 1}, H_{t}) = h (g ({\tilde{A}}_{t - 1}), f (H_{t}))

(2)

where $N R$ comprises an affine transformation $f$ applied to the node representation vector $H$ to learn abstract features. If more complex abstract representations are required, $f$ can be designed as an arbitrary multilayer feed-forward network or even a sophisticated deep-learning model. The function $g$ represents the variation function for the aggregation method of the connection representation matrix $A$ , and $h$ represents the activation function.

3). Connection Representation Module (CR):

The function of the CR module is to calculate the connection representation matrix based on the joint consideration of the current time step t’s input connection representation matrix (denoted as $A_{t}$ ) and node representation vector (denoted as ${\tilde{H}}_{t}$ ). Specifically, it extracts information from node features to guide the update of the connection representation matrix using the following procedure:

C R (A_{t}, {\tilde{H}}_{t}) = (F_{1} ({\tilde{H}}_{t}) + λ I) A_{t} + F_{2} ({\tilde{H}}_{t})

(3)

where $F_{1}$ and $F_{2}$ can represent any nonlinear function, and $λ$ represents the control coefficient that retains historical connection information.

From the descriptions of the three modules, it is evident that our proposed STCUF-EX exhibits strong versatility and scalability. It considers the relationships between temporal processing, node representation, and connection representation. It couples complementary spatial and temporal information while achieving good explainability through an alternative updating procedure.

B. STCUF-EX for Brain dFC Analysis

The brain network is naturally represented as a graph structure. Different brain regions correspond to distinct functions and can be regarded as nodes in the brain graph, while the interactions between different brain regions represent edges in the brain graph. We apply the STCUF-EX to brain dFC analysis, which is illustrated in Fig. 1.

1). Brain Network Definition and Data Initialization:

Inspired by graph theory, the complex brain network is partitioned into ROIs and modeled as a graph, where each ROI represents a node, and the correlation between the BOLD time series of specific ROIs are represented as edges. The dataset $G (V_{n}, E_{n})$ $(n = 1 : N)$ is defined as a weighted undirected graph $G$ , comprising $N$ subjects. For any $n$ , $V_{n}$ denotes a set of $M$ ROI nodes, and $E_{n}$ represents the set of edges connecting the $M$ ROIs, typically represented by an adjacency matrix $A_{n} \in ℝ^{M \times M}$ , describing the connectivity strength between ROIs.

The first step in the dFC analysis task is constructing the dFC matrix, for which we employ the classical sliding window technique [25], with a window width of $L$ and a step size of $g$ . Subsequently, the data after sliding is represented as ${{\hat{H}}_{t}}_{t = 1}^{T} \in ℝ^{M \times L}$ , where $M$ is the number of ROIs, $T$ is the total number of subwindows.

For each subwindow matrix, we first calculate the brain FC matrix $A_{t} = P C C ({\hat{H}}_{t})$ using the Pearson correlation and then normalize it to obtain ${\hat{A}}_{t} = D_{t}^{- \frac{1}{2}} (A_{t} + I) D_{t}^{- \frac{1}{2}}$ , where $D_{t} = \sum_{j} {(A_{t})}_{i j}$ , $I \in ℝ^{M \times M}$ .

In summary, we finally obtain an ordered set of brain region representation vectors and brain FC matrices, denoted as ${({\hat{H}}_{t}, {\hat{A}}_{t})}_{t = 1}^{T}$ . Next, we present the specific design details of spatio-temporal coupling learning.

2). Time Series Analysis-Based NR Module:

First, we construct a candidate brain node representation vector ${\tilde{H}}_{t}$ by incorporating spatial graph convolution operations to simulate the interactions between different brain regions in actual neural activity. Specifically, we dynamically update and include the FC matrix to avoid the potential graph over-smoothing that may arise from multiple graph convolutions using the same connectivity matrix:

{\tilde{H}}_{t} = \tanh (G C N_{H} + r_{t} ⊙ H_{t - 1}^{'} + b_{h})

(4)

in which $G C N_{H} : = relu (A_{t - 1}^{'} {\hat{H}}_{t} W_{H})$ represents the spatial convolution applied to the input information ${\hat{H}}_{t}$ at the current time step, learning the spatial correlations among brain regions to strengthen their connections. $H_{t - 1}^{'}$ represents the previous time step’s brain node’s state, $r_{t}$ denotes the reset gate, which controls the extent to which the previous time step’s state is written into the current candidate set, $⊙$ is the Hadamard product, and $b_{h}$ is the bias. The details are as follows:

r_{t} = σ (G C N_{r} + w_{r} H_{t - 1}^{'} + b_{r})

(5)

in which $G C N_{r} : = relu (A_{t - 1}^{'} {\hat{H}}_{t} W_{r})$ , $W_{r}$ and $w_{r}$ represents the learnable parameter matrices, $b_{r}$ is the bias, and $σ$ is the sigmoid function.

Finally, to maintain information propagation over long time sequences, the update gate $z_{t}^{'}$ controls the amount of past brain node features retained in the current time step, obtaining the updated brain node representation vector $H_{t}^{'}$ . The process is as follows:

z_{t}^{''} = σ (G C N_{z} + w_{z} H_{t - 1}^{'} + b_{z}) H_{t}^{'} = (1 - z_{t}^{'}) ⊙ H_{t - 1}^{'} + z_{t}^{'} ⊙ {\tilde{H}}_{t}

(6)

in which $G C N_{z} : = relu (A_{t - 1}^{'} {\hat{H}}_{t} W_{z})$ , $W_{z}$ and $w_{z}$ represent the learnable parameter matrices, $b_{z}$ is the bias, $σ$ is the sigmoid function, and $⊙$ is the Hadamard product.

3). Time Series Analysis-Based CR Module:

Considering the temporal dynamics of brain FC changes, we update in two steps: the computation of the candidate FC matrix and the preservation of historical functional connections. Firstly, the computation of the candidate FC matrix is as follows:

{\tilde{A}}_{t} = (S_{A T T} + λ I) ⊙ {\hat{A}}_{t}

(7)

where $λ$ represents the factor that preserves the initial FC information, and $S_{A T T} = (α_{i j})$ represents the attention scores. The computational process of $S_{A T T}$ is as follows.

Considering the influence of each node by itself and its neighboring nodes and recognizing that this influence is not fixed and may vary, we introduce an attention mechanism to enhance the relevant influence and better focus on significant connections between brain regions. To calculate the attention value $e_{i j}$ between node pairs $(i, j)$ , the influence of both nodes needs to be considered, and the computations are as follows:

e_{i j} = \tanh (W_{s} H_{t}^{'}) {(\tanh (V_{s} H_{t}^{'}))}^{T}

(8)

where $W_{s}$ and $V_{s}$ are trainable parameter matrices. Next, the attention scores for each node’s neighbors are normalized to obtain the final attention scores used in the spatio-temporal coupling model. The calculation is as follows:

α_{i j} = softmax (e_{i j}) = \frac{\exp (e_{i j})}{\sum_{k \in R_{i}} \exp (e_{k j})}

(9)

After the above steps, we obtain the candidate FC matrix, which allows us to utilize the brain node representation information to enhance the extraction of important connection information at the current time step. Furthermore, we consider preserving historically significant connection information by introducing a gate mechanism:

z_{t} = σ (W_{A} {\tilde{A}}_{t} + w_{A} A_{t - 1}^{'} + b_{A}) A_{t}^{'} = z_{t} ⊙ A_{t - 1}^{'} + (1 - z_{t}) ⊙ {\tilde{A}}_{t}

(10)

where $W_{A}$ and $w_{A}$ are learnable parameter matrices, $b_{A}$ is a bias term, $A_{t - 1}^{'}$ represents the FC matrix at the previous time step, and $z_{t}$ is the update gate, which controls the proportion of preserving historical connection information to the current connection information. It is evident that when $z_{t} = 0$ , historical connection information will be discarded entirely. $σ$ is the sigmoid function, and $⊙$ is the Hadamard product.

As a result, brain node representations can incorporate temporal information and aggregate spatial information for self-update across time steps. Moreover, a dynamic interaction update is performed on $H_{t}^{'}$ and $A_{t}^{'}$ at each time point, stimulating a brain-like spatio-temporal coupling mechanism.

III. Experimental Results and Analysis

This section uses STCUF-EX to explore the spatio-temporal evolution mechanisms of functional connections during brain development and the dFC changes in the brain during the resting state.

A. Data Acquisition and Preprocessing

The Philadelphia Neurodevelopmental Cohort (PNC) data comes from a large-scale collaborative study between the Brain Behavior Laboratory at the University of Pennsylvania and the Children’s Hospital of Philadelphia [26]. The dataset contains the rest fMRI data collected from 878 subjects aged 8 to 22 years. Among them, there are 193 children and 204 young adults. The details of the subjects are listed in Table I.

TABLE I.

Demographic characteristics of the subjects

	Children	Young Adults

Values	193	204
Gender(male/famale)	91/102	81/123
Age(Mean ± SD, month)	124.06 ± 11.33	231.50 ± 12.14

ASIAN	3(1.5%)	0(0%)
AFRICAN	77(39.9%)	74(36.3%)
AMERICAN	0(0%)	2(1%)
OTHER/MIXED	20(10.4%)	17(8.3%)
CAUCASIAN/WHITE	92(47.7%)	111(54.4%)
HAWAIIAN/PACIFIC	1(0.5%)	0(0%)

Open in a new tab

The data was acquired using a 3T Siemens TIM Trio scanner with a scanning duration of 6.2 minutes (124 TRs), and they were preprocessed using Statistical Parametric Mapping 12 (SPM12) for motion correction and spatial normalization. Additionally, a 3mm Full Width at Half Maximum (FWHM) Gaussian kernel was applied for data smoothing, and bandpass filtering was performed within the frequency range of 0.01–0.1Hz on the functional time series. We employed a 264-ROI atlas, utilizing a sphere radius parameter of 5 mm to perform dimensionality reduction on the dataset [27]. The signals from different voxels in an ROI were averaged, resulting in a 264 × $T_{s}$ matrix for each subject, where $T_{s}$ corresponds to a total of 124-time points for each rs-fMRI scan, and the time interval is 3s.

We use the sliding-window technique to estimate the dFC. For this method, a window with length $L$ is moved along the time series by step size $g$ , with subsequent computation of the dFC between two ROIs using the Pearson correlation coefficient (PCC) for each window. In total, $(T = [(T_{s} - L) / g] + 1)$ sliding window for $T_{s}$ time points of the time series. To select parameters more suitable for the model, we iterated different window widths within the range of 30–60 seconds (corresponding to 10–20 $TRs$ ), as well as different sliding step lengths (2:2:10 $TRs$ ) and fixed other model parameters for training. Finally, the sliding window parameters corresponding to the highest classification accuracy on the training data are chosen as the ultimate selection. The specific experimental results are shown in Fig. 2, yielding $L = 12 T R s$ and $g = 4 T R s$ , where 1 $TR$ equals 3 seconds. As a result, the entire BOLD time series signal, comprising 124 time points, was segmented into 29 distinct subsequences. Based on this, the PCC between two ROIs is calculated for each subsequence, and the data processed is represented as ${X_{t}}_{t = 1}^{T} \in ℝ^{M \times L}$ , where $M$ denotes the number of ROI, as mentioned above, we have $T = 29$ and $L = 12$ . Finally, the data is represented as ${X_{t}}_{t = 1}^{29} \in ℝ^{264 \times 12}$ .

Fig. 2. — The influence of parameter in sliding window on the classification accuracy in the PNC data.

For all participants, we randomly selected 75% (297) of samples from each group as the training data to identify significant differences in dFC between the two groups. The remaining 25% (100) of samples were used as validation data to evaluate the selected dFC features.

B. Experimental Results

1). Parameter setting:

The training data was further divided to determine the hyperparameters, with 70% of the training data used for model training and 30% for evaluating the hyperparameters. The grid search method was employed to select suitable hyperparameters, as it allows a complete search of the given hyperparameter space, which is easily parallelized and leads to more stable optimal hyperparameters [28]. Specifically, each hyperparameter was selected using the grid search method while keeping other hyperparameters fixed. By repeating this process, all hyperparameters were determined. After grid search, the hyperparameters were set as follows: epochs = 70, learning rate = 0.001, and dropout rate = 0.2. All experiments in this study were conducted using PyTorch 1.4.0, and the AdamW optimizer was used for model training.

2). Significant difference connection selection:

Based on the output results $H_{t}^{'}$ from the final time step, we perform classification for children and young adults by using a two-layer multilayer perceptron. In addition, a feature selection layer was added after the output layer to obtain the most significant difference of FC between children and young adults. Following the approach of Xu et al. [25], a two-sample t-test with a significance level of p = 0.05 was used to test for significant differences. The most discriminative features were then selected, and a support vector machine was employed to calculate the classification accuracy of the selected features on the test data to assess their discriminative ability. The optimal set of 2461 features was selected from the original 34,980 features, leading to dFC for subsequent analysis.

3). Model performance analysis:

To demonstrate the effectiveness of our proposed method, we conducted comparative experiments with a series of classical methods and the proposed method. The selected methods include traditional classical methods: support vector machine (SVM), logistic regression (LR), decision tree (DT), k-nearest-neighbor (KNN), and classic deep learning methods: graph convolutional network (GCN), graph attention network (GAT), recurrent neural network (RNN), long short-term memory (LSTM), and gated recurrent unit (GRU), spatio-temporal graph convolutional network (STGCN), dynamic adaptive spatio-temporal graph convolution (DAST-GCN), multi-channel spatio-temporal graph convolutional network (MCST-GCN), anatomy-guided spatio-temporal graph convolutional networks (AG-STGCN), pedestrian trajectory prediction model based on a spatio-temporal graph convolutional network (PTP-STGCN). Four performance metrics were chosen to evaluate the performance of each model, including classification accuracy (ACC), precision, recall, and F1-score. For a fair comparison, all deep graph models are composed of 2 layers and trained under the same conditions. After grid search, the hyperparameters were set for each model. For SVM, we set the maximum iteration number = 64, and the kernel function is linear. For LR, we set the maximum iteration number = 64, the kernel function is sigmoid, and the learning rate = 0.001. For DT, we set the criterion = ‘gini’ and max depth = 10. For KNN, we set $k = 5$ . For GCN, we set the hidden layer dimension = 33, learning rate = 0.0005, weight decay = 5e-5, and dropout = 0.2. For GAT, we set the hidden layer dimension = 33, learning rate = 0.004, dropout rate = 0.4, and alpha = 0.2. For RNN, we set the dropout rate = 0.2, hidden size = 33, num layers = 2, and nonlinearity = ‘relu’. For LSTM, we set the dropout rate = 0.2, hidden size = 33, and num layers = 2. For GRU, we set the dropout rate = 0.2, hidden size = 33, and num layers = 2. For STGCN, we set the maximum iteration number = 64, starter learning rate = 0.1, learning rate exp = 0.1, window size = 12, learning rate step = 1000, and batch size = 30. For DAST-GCN, we set the initial learning rate = 0.008, the number of spatiotemporal blocks as 3, the embedding dimension of the node dictionaries as 10, the number of filters as 10, the kernel size as 3, weight decay = 0.005, and the dropout rate = 0.4. For OT-MCSTGCN, we set the learning rate = 0.004 and weigh decay = 0.001. In the temporal convolution block, the size of each convolution kernel is 3*3, and the stride is 1. In the spatial convolution block, the size of the graph convolution kernel is 45, and the order of the Chebyshev polynomial is set to be 3. For AG-STGCN, we set the length of the spatial kernel as 10, the length of the temporal kernel is 4, the initialized learning rate is 0.005, and the weight decay rate is 0.002. For PTP-STGCN, we set the learning rate as 0.0045, the SGCN with a kernel size of three, the T-transformer consisting of eight-head encoding layers, the S-GCN with the embedding vector dimension size of six, and the T-transformer with embedding vector dimension of 40 sequentially by the fully-connected layers. Each comparative model underwent five random experiments with the optimal number of layers and parameters, and the average performance results are presented in Table II. Also, we conducted ablation experiments to verify the influence of the attention mechanism on model learning. The results are presented in Table III. The comparison of the performance of $T A$ based on LSTM and GRU is presented in Table IV.

TABLE II.

Comparative Results of Various Baseline Models for Classifying Children and Young Adults in PNC Data

Method	ACC	Recall	Precision	F1-Score

SVM	0.5015 ± 0.0184	0.4976 ± 0.0194	0.5256 ± 0.0304	0.5133 ± 0.0276
LR	0.5480 ± 0.0129	0.5440 ± 0.0123	0.5680 ± 0.0191	0.5595 ± 0.0170
DT	0.6225 ± 0.0152	0.6980 ± 0.0122	0.4820 ± 0.0136	0.5669 ± 0.0861
KNN	0.5250 ± 0.0534	0.4752 ± 0.0094	0.5572 ± 0.3004	0.5121 ± 0.1943
GCN	0.5654 ± 0.0279	0.5347 ± 0.1031	0.5451 ± 0.0392	0.5329 ± 0.0400
GAT	0.6150 ± 0.0100	0.5770 ± 0.0592	0.6302 ± 0.0151	0.6173 ± 0.0259
RNN	0.6421 ± 0.0203	0.5510 ± 0.0790	0.5934 ± 0.0273	0.5712 ± 0.0386
LSTM	0.6545 ± 0.0237	0.6327 ± 0.0470	0.6960 ± 0.0192	0.6458 ± 0.0353
GRU	0.6549 ± 0.0229	0.6122 ± 0.0496	0.6829 ± 0.0227	0.6316 ± 0.0335
STGCN	0.7621 ± 0.0142	0.7736 ± 0.0442	0.7013 ± 0.0264	0.7356 ± 0.0330
DAST-GCN	0.7533 ± 0.0373	0.7336 ± 0.0373	0.7517 ± 0.0341	0.7414 ± 0.0460
MCSTGCN	0.7719 ± 0.0281	0.7570 ± 0.0428	0.7692 ± 0.0283	0.7626 ± 0.0316
AG-STGCN	0.7643 ± 0.0546	0.7856 ± 0.0435	0.7765 ± 0.0566	0.7867 ± 0.0656
PTP-STGCN	0.7875 ± 0.0456	0.7754 ± 0.0343	0.7887 ± 0.7887	0.7745 ± 0.0894
STCUF-EX	0.8131 ± 0.0110	0.8305 ± 0.0433	0.7850 ± 0.0241	0.8157 ± 0.0338

Open in a new tab

TABLE III.

The ablation experiment for the influence of attention mechanisms on model learning

Method	ACC	Recall	Precision	F1-Score

STCUF-EX without ATT	0.8002 ± 0.0833	0.7778 ± 0.0746	0.8045 ± 0.1137	0.7907 ± 0.0924
STCUF-EX	0.8131 ± 0.0110	0.8305 ± 0.0433	0.7850 ± 0.0241	0.8157 ± 0.0338

Open in a new tab

TABLE IV.

Comparative results of the $TA$ model based on LSTM and GRU respectively for classifying children and young adults

Method	ACC	Recall	Precision	F1-Score

STCUF-EX-LSTM	0.7817 ± 0.0645	0.7454 ± 0.0834	0.7952 ± 0.0770	0.7676 ± 0.0700
STCUF-EX	0.8131 ± 0.0110	0.8305 ± 0.0433	0.7850 ± 0.0241	0.8157 ± 0.0338

Open in a new tab

From the above table, our proposed method has significantly improved the classification accuracy compared to the traditional machine learning models SVM, LR, DT, and KNN by 31.16%, 26.51%, 19.06%, and 28.81%, respectively. Furthermore, compared to the classical deep learning models GCN, GAT, RNN, LSTM, GRU, STGCN, DAST-GCN, MCSTGCN, AG-STGCN, and PTP-STGCN, our method has shown significant improvements in classification accuracy by 24.77%, 19.81%, 17.10%, 15.86%, 15.82%, 5.10%, 5.98%, 4.12%, 4.88% and 2.56% respectively. Additionally, all other performance metrics have demonstrated remarkable improvement over the various models we compared. These results demonstrate the superiority of our proposed method, providing support for the subsequent analysis. Meanwhile, the results of the ablation experiments indicate that the introduction of the attention mechanism also contributes to the improvement of classification performance. The comparative experiments of $TA$ based on LSTM and GRU respectively indicate that for the $TA$ model with fewer parameters and lower computational costs is more suitable for the analysis of time-series data.

C. DFC Analysis

We further grouped the 264 ROIs into 13 resting-state networks (RSNs) based on their functionalities to better analyze the FC between ROIs [26]. They are sensory/somatomotor network (SSN), cingulo-opercular task control network (COTCN), fronto-parietal task control network (FPTCN), auditory network (AN), default mode network (DMN), memory retrieval network (MRN), visual network (VN), salience network (SN), subcortical network (SCN), ventral attention network (VAN), dorsal attention network (DAN), cerebellar network (CN), and uncertain network (UN).

Next, we mapped the FC matrices obtained at each time step to the 13 RSNs. Considering the symmetry of the connectivity matrices, we selected the lower triangular elements and arranged them column-wise into vectors for further analysis. As a result, each sample was transformed into 29 vectors of length 91, and we selected the middle 27 vectors for analysis to ensure the effectiveness of the data.

To describe the time-varying FC differences between children and young adults, we employed k-means clustering to identify different patterns of FC that repeatedly showed up in the subjects, referred to as dFC states. The number of clusters was determined using the silhouette coefficient maximization method. As a result, the optimal number of clusters for children and young adults was 4 and 5, respectively. The dFC states for children were labeled as state I, state II, state III, and state IV, while the dFC states for young adults were labelled as state I’, state II’, state III’, state IV’, and state V’. The proportions of the four dFC states for children were 60.53%, 37.16%, 1.36%, and 0.95%, respectively. For young adults, the proportions of the five dFC states were 55.93%, 39.96%, 0.91%, 0.91%, and 2.30%, respectively.

To reveal the dynamic changes in FC throughout the scanning process, we first present the variations of FC strengths in children and young adults for each state from a three-dimensional perspective, as shown in Fig. 3; it is evident that both children and young adults exhibit relatively stable and nonfluctuating FC strengths in their respective state 1. In state 2, they both demonstrate consistent and stable patterns of changes in FC strengths, showing a high degree of similarity between the two groups. On the other hand, more significant fluctuations are observed in other states.

Fig. 3. — The results of analyzing dFC states in children and young adults. (a) Three-dimensional plot of FC strengths for each state in children. (b) Three-dimensional plot of FC strengths for each state in young adults.

Furthermore, we compared State 2 between children and young adults, as shown in Fig. 4(a-b). There is enhanced FC between the SSN and RSNs such as FPTCN, VN, UN, and DMN. On the other hand, there is a certain degree of weakening in connections between DMN and VAN and DAN. Additionally, regarding the changes in connectivity strengths within RSNs, significant weakening is observed in the DMN, followed by relatively noticeable weakening in RSNs such as SSN, VN, FPTCN, and UN. The fluctuations in connectivity between DMN and other networks reflect that during the resting state, participants still have intrinsic mental activities, as evidenced by the involvement of the DMN. Moreover, we found that these weakened functional connections exhibit opposite results in states with small proportions. In those states, young adults demonstrate enhanced FC compared to their counterparts in children.

Next, we calculated the proportions of children and young adults in each state throughout the scanning process, and the specific results are displayed in Fig. 4(c-d). The experimental results indicate that during the entire scanning process, most brain FC states remain primarily in state I and II for children, while the stay in states III and IV is exceptionally brief. As for the brain, FC states of young adults are mainly concentrated in state I’ and state II’, with very brief stays in state III’, state IV’, and state V’. Notably, the proportions of children and young adults in each state exhibit a regular pattern of state changes in the resting-state, consistent with existing research findings that suggest the presence of regular functional activities in the resting-state brain [29].

Furthermore, to investigate the time occupied divergence of each subject in different states, we calculated the dwell time (DT) and fraction time (FT) for 193 children and 204 young adults based on their respective state transition vectors. Additionally, we calculated the mean dwell time (MDT) and mean fraction time (MFT) for each state. Specifically,

D T^{s t a t e (k)} = count (difference (i, k) = = - 1) - count (difference (i, k) = = 1) F T^{s t a t e (k)} = \frac{count ({state_vector}_{(i)} = = k)}{Total number of T R}

(11)

where ”1” and ”−1” indicate that the i-th sample belongs to a certain state $k$ or not, ${state_vector}_{(i)}$ represents the state of the $i$ -th sample, and $T R$ is the repetition time [30].

In Fig. 5, the dots represent the DT or FT values of all participants in each state. The rectangular blocks represent the boxplots of the DT or FT values of all participants in each state. The lines in the graph connect the mean values obtained from different states. Children spend more time in states I and II, while young adults spend more time in states I’ and II’. Additionally, both children and young adults rarely stay in the other respective states. This finding is consistent with the conclusions drawn from our previous analysis.

D. Differential Brain FC Analysis

For the selected 2461 significant functional connections associated with developmental differences, following the approach of Qiao et al. [19], we identified 1161 connections that significantly increased with age and 1300 connections that significantly decreased. Fig. 6 displays the distribution of FC differences at both the ROI and RSN levels. For the selected 2461 connections considered significantly associated with developmental differences, we employed BrainNet Viewer for 3-D visualization [31], as shown in Fig. 6 (a). Different colored nodes represent different ROIs, and the thickness of the connections represents the strength of connectivity between different ROIs.

Fig. 6 (b) presents the distribution of these differential connections in a two-dimensional plane. Furthermore, we mapped these significant connections to the 13 RSNs, as shown in Fig. 6 (c)(d), which illustrates the functional connections that significantly increase or decrease in adults compared to children. At the population level, the degree of enhancement in FC is less than the degree of weakening as individuals mature.

The visualization results show that the differential functional connections significantly increased in adults compared to children are mainly concentrated between the UN and VN, the DMN and SSN, the SN and COTCN, the CN and VN, and between the FPTCN and VN. Meanwhile, the differential functional connections significantly decreased in adults compared to children and are mainly concentrated between the DAN and VAN, the SCN and AN, the SN and DMN, the AN and MRN, the DAN, SN, and SSN with the VAN, and between the SCN and SN and between the COTCN and FPTCN.

IV. DISCUSSION

In this study, we proposed an explainable framework coupling spatio-temporal information (STCUF-EX) and applied it to investigate the dFC changes and differences between children and young adults in rest-fMRI data. These functional connections were mainly observed within and between the DMN, VN, FPTCN, SN, SSN, AN, and SCN. These RSNs contain distinguishable information closely associated with information processing, attention, alertness, cognition, and working memory functions.

Our study results indicate that with brain development, there are relatively more regions showing weakened FC, especially within the DMN and between the DMN and other RSNs (such as SCN, SN, FPTCN, AN, VN, COTCN, VAN, etc.). Young adults exhibit a gradual reduction in FC within the DMN compared to children, a finding consistent with previous biological research [32]. Broyd et al. found that the connections related to the DMN were stronger in the brain network of children aged 9–12 [30]. Similarly, Hutchison et al. concluded that children had stronger connectivity between the DMN and other networks than young adults [33]. The DMN is mainly associated with intrinsic mental activities such as memory [34]. The reduced connectivity between the DMN and other networks in young adults compared to children suggests that the functional networks of children may be less specialized or effective in information processing, indicating that there may be redundant connections in the brain during the processing of complex cognitive or emotional processes [35]. Regarding the connections between different RSNs, we found weakened FC between the DMN and SCN with age. Studies have shown that the reduced connectivity associated with the SCN indicates a general decrease in cortical FC during development [36]. We also observed weakened FC between the DMN and SN with age. Previous research has also shown that the increase in DMN and SN activity is related to more defensive brain tissues in the heterogenous static interoceptive brain system [37]. Therefore, our findings suggest that during the resting state, the coordination between the DMN and SN in children is stronger than in young adults, indicating that children exhibit more defensive brain tissues than young adults. Additionally, our results demonstrate weakened FC between the DMN and FPTCN with age. Previous research has found that weakening connections between the DMN and FPTCN are associated with higher reading abilities during development [38]. Furthermore, the connectivity strength between the DMN and FPTCN is negatively correlated with higher-order cognitive abilities during development [39]. Our findings also indicate weakened FC between the DMN and AN. Previous research has shown causal interaction loops between the auditory and default modes networks. The reduced response of the auditory cortex also leads to a decrease in the inhibitory ability of the auditory network on the DMN through asynchronous interactions between regions [40]. It may indicate that young adults are less sensitive to environmental sounds than children. Additionally, we found weakened FC between the DMN and VN with age. This finding is worthy of further research. Finally, our results show weakened FC between the DMN and COTCN with age. Previous studies have indicated that specific ROI in the COTCN is primarily responsible for information transmission between networks, and the network is highly active in performing many complex cognitive tasks [41]. Some studies have shown that with increasing age during adolescence and early adulthood, the connectivity between the DMN and COTCN decreases [42]. Furthermore, the connectivity between the FPTCN and the SSN, as well as the AN in young adults, also exhibited significant weakening. It can be attributed to FPTCN’s involvement in tasks related to selective attention control, working memory maintenance, predictive perceptual coding, and cognitive processes [43]. Completing such cognitive tasks during childhood primarily relies on auditory, somatosensory, and task-related stimuli without a fully developed independent thinking system. In contrast, adults demonstrate a more mature cognitive system, showing reduced reliance on external conditions such as auditory and task stimuli, and possess a clearer sense of subjective judgment. This finding aligns with the earlier proposition by Biao Cai et al. [43]. Additionally, previous research has indicated weakened connectivity between FPTCN and SSN with age, while children exhibit stronger connectivity between these two RSNs [30]. This phenomenon is primarily attributed to the gradual formation of complex cognitive processes with age, leading to the presence of redundant connections in the brain networks of children. Thus, greater FC exists during childhood, with relatively lower connectivity between hierarchically organized functional groups [44].

In addition, compared to children, the significantly enhanced differential FC in young adults primarily concentrates between the SSN and the UN, VN and DMN; between the CN and the SN, as well as the COTCN, and between VN and FPTCN and the SCN. These findings align well with numerous existing conclusions. Prior studies have indicated that with brain development, brain connectivity associated with SSN strengthens with age [30]. The enhanced FC between SSN and VN has been validated in the literature, suggesting that young adults have higher visual information processing capabilities than children [45]. FPTCN can initiate and adjust top-down control, and its enhanced FC represents the progressive refinement of cognitive and motor abilities during childhood development, leading to more sophisticated functional networks [25]. The strengthened FC between FPTCN and VN with age indicates that adults generally exhibit higher visual information processing capabilities and cognitive control than children. The enhanced FC between COTCN and CN is believed to form independent subnetworks related to inter-task error-related activity involving transferring task-control information and feedback to control networks [46]. Moreover, the enhanced dFC between SCN and VN suggests that young adults possess superior task-processing abilities compared to children.

Furthermore, Fig. 2–4 show that the patterns of FC within RSN change between children and young adults throughout the entire state transition process. It is evident that both children and young adults primarily switch between their state 1 (I) and state 2 (II) during the entire scanning procedure. Previous analyses have shown that state 1 for children and young adults is characterized by relative stability without significant fluctuations and constitutes the largest proportion in all their respective states. On the other hand, state 2 accounts for the second-largest proportion, and the FC changes between these two states are highly similar. Additionally, considering the regularity of brain functional network states observed in the percentage distribution of children and young adults across different states during rest, our research reveals a shared pattern and similar trends in resting-state changes between children and young adults. Specifically, during rest, brain functional connections transition from a relatively stable state to another activated state, and these two states undergo periodic alternation. With small proportions, the remaining states can be understood as fluctuations induced by external or psychological influences during the scanning process. Notably, in all fluctuating states, the DMN exhibits the most prominent changes in FC, which aligns with existing research conclusions that during rest, subjects still engage in internal psychological activities and reflect them through DMN [47]. The DMN functional regions are essential parts of the brain that exhibit high neural activity and energy consumption during rest [34]. During the state transition process, SSN, VN, and UN also demonstrate significant changes, indicating that compared to children, young adults’ brain activity between cognition and emotion-related RSNs is more frequent during rest. Specifically, the more active the brain activity, the stronger the functional coordination. The discussions above reveal dynamic differences in brain activity during rest as one grows and shed light on potential reasons for brain state transitions.

In summary, our research results indicate that with increasing age, complex cognitive processes gradually develop, leading to some redundant connections in the brain networks of children, resulting in lower FC between hierarchical functional organizations [32]. There is also a tendency for FC to weaken during brain development, indicating that children exhibit more dispersed FC patterns. In contrast, young adults demonstrate more integrated FC patterns [48]. In other words, as the brain matures, FC shifts from segregation to integration. It suggests that brain organization can more efficiently transmit information as one grows and brain functions transition from undifferentiated systems to specialized networks [49]. Furthermore, we found two developmental trends in FC during growth. Connectivity between regions associated with emotional regulation will weaken, and each region will become more specialized. In contrast, connectivity between regions involved in cognitive activities will strengthen, and adults tend to have better within-network connectivity, while children exhibit stronger between-network connectivity [32]. Additionally, children and young adults show two similar primary state patterns during rest, exhibiting regular and periodic alternations.

V. Conclusion

In this paper, we present an explainable deep learning framework for spatio-temporal information coupling, namely, STCUF-EX. It can effectively uncover the coupling relationships within spatio-temporal data and further explore their evolution mechanisms. Specifically, this framework dynamically learns the coupling of node/ROI representation and inter-node connectivity as they change over time, incorporating potential spatio-temporal correlations within the data. At the end, we apply STCUF-EX to the study of dFC, showing significant differences in brain development as well as the changes in the evolution patterns of the brain within the resting state. Using the framework, we find that young adults exhibit more clustered FC patterns during growth than children, and FC shows two distinct developmental trends. In other words, the connectivity between regions associated with emotional regulation weakens, while the connectivity between regions linked to cognitive activities strengthens. Additionally, children and young adults exhibit significant periodic variations in brain FC in the resting state.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (Nos. 12090021, 12090020, 12271429, and 12226007), the Natural Science Basic Research Program of Shaanxi (No. 2022JM005), the National Key Research and Development Program of China (No. 2020AAA0106302), and was partly supported by the National Institutes of Health (R01 MH104680, R01 GM109068, R01 MH121101, R01 MH116782, R01 MH118013 and P20-GM144641) and the HPC Platform, Xi’an Jiaotong University.

Contributor Information

Bin Gao, School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, P.R. China.

Aiju Yu, School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, P.R. China.

Chen Qiao, School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, P.R. China.

Vince D. Calhoun, Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA 30303.

Julia M. Stephen, Mind Research Network, Albuquerque, NM 87106.

Tony W. Wilson, Institute for Human Neuroscience, Boys Town National Research Hospital, Boys Town, NE 68010.

Yu-Ping Wang, Department of Biomedical Engineering, Tulane University, New Orleans, LA 70118 USA.

References

[1].Calhoun VD, Miller R, Pearlson G, and Adalı T, “The chronnectome: Time-varying connectivity networks as the next frontier in fMRI data discovery,” Neuron, vol. 84, no. 2, pp. 262–274, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
[2].Ferreira LN et al. , “Spatiotemporal data analysis with chronological networks,” Nature Communications, vol. 11, no. 1, Aug. 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
[3].Huang H et al. , “Modeling Task fMRI Data Via Deep Convolutional Autoencoder,” IEEE Transactions on Medical Imaging, vol. 37, no. 7, pp. 1551–1561, Jul. 2018. [DOI] [PubMed] [Google Scholar]
[4].Zhang W et al. , “Hierarchical Organization of Functional Brain Networks Revealed by Hybrid Spatiotemporal Deep Learning,” Brain Connectivity, vol. 10, no. 2, pp. 72–82, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
[5].Zhao Y et al. , “Four-dimensional modeling of fMRI data via spatio–temporal convolutional neural networks (ST-CNNs),” IEEE Transactions on Cognitive and Developmental Systems, vol. 12, no. 3, pp. 451–460, 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
[6].Xiao J et al. , “A spatio-temporal decomposition framework for dynamic functional connectivity in the human brain,” NeuroImage, vol. 263, p. 119618, 2022. [DOI] [PubMed] [Google Scholar]
[7].Yan J et al. , “Multi-head GAGNN: A multi-head guided attention graph neural network for modeling spatio-temporal patterns of holistic brain functional networks,” Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, pp. 564–573, 2021.
[8].El-Gazzar A, Thomas RM, and Wingen GV, “Dynamic adaptive spatio-temporal graph convolution for fMRI modeling,” Machine Learning in Clinical Neuroimaging, pp. 125–134, Sep. 2021.
[9].Ye Z, Qu Y, Liang Z, Wang M, and Liu Q, “Explainable fMRI-based brain decoding via spatial temporal-pyramid graph convolutional network,” Human Brain Mapping, vol. 44, no. 7, pp. 2921–2935, Feb. 2023. [DOI] [PMC free article] [PubMed] [Google Scholar]
[10].Zhu Q et al. , “Spatio-Temporal Graph Hubness Propagation Model for Dynamic Brain Network Classification,” IEEE Transactions on Medical Imaging, p. 1, Jan. 2024. [DOI] [PubMed]
[11].Azevedo T et al. , “A deep graph neural network architecture for modelling spatio-temporal dynamics in resting-state functional MRI data,” Medical Image Analysis, vol. 79, p. 102471, Jul. 2022. [DOI] [PubMed] [Google Scholar]
[12].Huang S-G, Xia J, Xu L, and Qiu A, “Spatio-temporal directed acyclic graph learning with attention mechanisms on brain functional time series and connectivity,” Medical Image Analysis, vol. 77, p. 102370, Apr. 2022. [DOI] [PubMed] [Google Scholar]
[13].Kong Y et al. , “Spatio-temporal graph convolutional network for diagnosis and treatment response prediction of major depressive disorder from functional connectivity,” Human Brain Mapping, vol. 42, no. 12, pp. 3922–3933, May 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
[14].Yin Y, Zheng X, Hu B, Zhang Y, and Cui X, “EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM,” Applied Soft Computing, vol. 100, p. 106954, Mar. 2021. [Google Scholar]
[15].Lian J, Ren W, Li L, Zhou Y, and Zhou B, “PTP-STGCN: Pedestrian Trajectory Prediction Based on a Spatio-temporal Graph Convolutional Neural Network,” Applied Intelligence, vol. 53, pp. 2862–2878, May. 2022. [Google Scholar]
[16].Jiang M et al. , “Anatomy-Guided Spatio-Temporal Graph Convolutional Networks (AG-STGCNs) for Modeling Functional Connectivity Between Gyri and Sulci Across Multiple Task Domains,” IEEE transactions on neural networks and learning systems, pp. 1–11, Jan. 2022. [DOI] [PubMed]
[17].Talukder A, Barham C, Li X, and Hu H, “Interpretation of deep learning in genomics and epigenomics,” Briefings in Bioinformatics, vol. 22, no. 3, pp. 1–16, Aug. 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
[18].Saeed W and Omlin C, “Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities,” Knowledge-Based Systems, vol. 263, p. 110273, Jan. 2023. [Google Scholar]
[19].Qiao C et al. , “Deep learning with explainability for characterizing age-related intrinsic differences in dynamic brain functional connectivity,” Medical Image Analysis, vol. 90, p. 102941, Dec. 2023. [DOI] [PubMed] [Google Scholar]
[20].Barredo Arrieta A et al. , “Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI,” Information Fusion, vol. 58, no. 1, pp. 82–115, Jun. 2020. [Google Scholar]
[21].Guidotti R, Monreale A, Ruggieri S, Turini F, Giannotti F, and Pedreschi D, “A Survey of Methods for Explaining Black Box Models,” ACM Computing Surveys, vol. 51, no. 5, pp. 1–42, Aug. 2018. [Google Scholar]
[22].Allen EA, Damaraju E, Plis SM, Erhardt EB, Eichele T, and Calhoun VD, “Tracking Whole-Brain Connectivity Dynamics in the Resting State,” Cerebral Cortex, vol. 24, no. 3, pp. 663–676, Nov. 2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
[23].McIntosh AR, “Towards a network theory of cognition,” Neural Networks, vol. 13, no. 8–9, pp. 861–870, Nov. 2000. [DOI] [PubMed] [Google Scholar]
[24].Gazzaniga MS, Ivry RB, and Mangun GR, Cognitive Neuroscience the Biology of Mind. New York, NY u.a: Norton, 2002.
[25].Xu F et al. , “An explainable autoencoder with multi-paradigm fMRI fusion for identifying differences in dynamic functional connectivity during brain development,” Neural Networks, vol. 159, pp. 185–197, Feb. 2023. [DOI] [PMC free article] [PubMed] [Google Scholar]
[26].Zille P, Calhoun VD, Stephen JM, Wilson TW, and Wang Y-P, “Fused estimation of sparse connectivity patterns from Rest fMRI-Application to comparison of children and Adult Brains,” IEEE Transactions on Medical Imaging, vol. 37, no. 10, pp. 2165–2175, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
[27].Power JD et al. , “Functional network organization of the human brain,” Neuron, vol. 72, no. 4, pp. 665–678, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
[28].Haqmi Abas MA, “Agarwood Oil Quality Classification using Support Vector Classifier and Grid Search Cross Validation Hyperparameter Tuning,” International Journal of Emerging Trends in Engineering Research, vol. 8, no. 6, pp. 2551–2556, Jun. 2020. [Google Scholar]
[29].Biswal B, Zerrin Yetkin F, Haughton VM, and Hyde JS, “Functional connectivity in the motor cortex of resting human brain using echoplanar MRI,” Magnetic Resonance in Medicine, vol. 34, no. 4, pp. 537–541, Oct. 1995. [DOI] [PubMed] [Google Scholar]
[30].Cai B et al. , “Estimation of dynamic sparse connectivity patterns from resting state fMRI,” IEEE Transactions on Medical Imaging, vol. 37, no. 5, pp. 1224–1234, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
[31].Stark CEL and Squire LR, “When zero is not zero: The problem of ambiguous baseline conditions in fMRI,” Proceedings of the National Academy of Sciences, vol. 98, no. 22, pp. 12760–12766, Oct. 2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
[32].Zhang G et al. , “Estimating Dynamic Functional Brain Connectivity With a Sparse Hidden Markov Model,” IEEE Transactions on Medical Imaging, vol. 39, no. 2, pp. 488–498, Feb. 2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
[33].Edin F, Macoveanu J, Olesen P, Tegnér J, and Klingberg T, “Stronger Synaptic Connectivity as a Mechanism behind Development of Working Memory-related Brain Activity during Childhood,” Journal of Cognitive Neuroscience, vol. 19, no. 5, pp. 750–760, May 2007. [DOI] [PubMed] [Google Scholar]
[34].Raichle ME, MacLeod AM, Snyder AZ, Powers WJ, Gusnard DA, and Shulman GL, “A default mode of brain function,” Proceedings of the National Academy of Sciences, vol. 98, no. 2, pp. 676–682, Jan. 2001. [DOI] [PMC free article] [PubMed] [Google Scholar]
[35].Cai B et al. , “Capturing Dynamic Connectivity From Resting-State fMRI Using Time-Varying Graphical Lasso,” IEEE Transactions on Biomedical Engineering, vol. 66, no. 7, pp. 1852–1862, Jul. 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
[36].Allen EA et al. , “A Baseline for the Multivariate Comparison of Resting-State Networks,” Frontiers in Systems Neuroscience, vol. 5, p. 2, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
[37].Kozlowska K et al. , “‘Motoring in idle’: The default mode and somatomotor networks are overactive in children and adolescents with functional neurological symptoms,” NeuroImage: Clinical, vol. 18, pp. 730–743, 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
[38].Jolles DD, Mennigen E, Gupta MW, Hegarty CE, Bearden CE, and Karlsgodt KH, “Relationships between intrinsic functional connectivity, cognitive control, and reading achievement across development,” NeuroImage, vol. 221, p. 117202, Nov. 2020. [DOI] [PubMed] [Google Scholar]
[39].Briley PM et al. , “Development of human electrophysiological brain networks,” Journal of Neurophysiology, vol. 120, no. 6, pp. 3122–3130, Dec. 2018. [DOI] [PMC free article] [PubMed] [Google Scholar]
[40].Xu L-C et al. , “Abnormal neural activities of directional brain networks in patients with long-term bilateral hearing loss,” Oncotarget, vol. 8, no. 48, pp. 84168–84179, Aug. 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
[41].Sheffield JM et al. , “Fronto-parietal and cingulo-opercular network integrity and cognition in health and schizophrenia,” Neuropsychologia, vol. 73, pp. 82–93, Jul. 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
[42].Anderson JS, Ferguson MA, Lopez-Larson M, and Yurgelun-Todd D, “Connectivity Gradients Between the Default Mode and Attention Control Networks,” Brain Connectivity, vol. 1, no. 2, pp. 147–157, Aug. 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
[43].Broyd SJ, Demanuele C, Debener S, Helps SK, James CJ, and Sonuga-Barke EJS, “Default-mode brain dysfunction in mental disorders: A systematic review,” Neuroscience & Biobehavioral Reviews, vol. 33, no. 3, pp. 279–296, Mar. 2009. [DOI] [PubMed] [Google Scholar]
[44].Supekar K, Musen M, and Menon V, “Development of Large-Scale Functional Brain Networks in Children,” NeuroImage, vol. 47, p. S109, Jul. 2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
[45].Bostan AC, Dum RP, and Strick PL, “Cerebellar networks with the cerebral cortex and basal ganglia,” Trends in Cognitive Sciences, vol. 17, no. 5, pp. 241–254, May 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
[46].Fair DA, Cohen AL et al. , “Functional Brain Networks Develop from a ‘Local to Distributed’ Organization,” PLoS Computational Biology, vol. 5, no. 5, p. e1000381, May 2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
[47].Spreng RN, Mar RA, and Kim ASN, “The Common Neural Basis of Autobiographical Memory, Prospection, Navigation, Theory of Mind, and the Default Mode: A Quantitative Meta-analysis,” Journal of Cognitive Neuroscience, vol. 21, no. 3, pp. 489–510, Mar. 2009. [DOI] [PubMed] [Google Scholar]
[48].Kelly AMC, Di Martino A et al. , “Development of Anterior Cingulate Functional Connectivity from Late Childhood to Early Adulthood,” Cerebral Cortex, vol. 19, no. 3, pp. 640–657, Mar. 2009. [DOI] [PubMed] [Google Scholar]
[49].Jolles DD, van Buchem MA, Crone EA, and Rombouts SARB, “A Comprehensive Study of Whole-Brain Functional Connectivity in Children and Young Adults,” Cerebral Cortex, vol. 21, no. 2, pp. 385–391, 2011. [DOI] [PubMed] [Google Scholar]

PERMALINK

An Explainable Unified Framework of Spatio-Temporal Coupling Learning with Application to Dynamic Brain Functional Connectivity Analysis

Bin Gao

Aiju Yu

Chen Qiao

Vince D Calhoun

Julia M Stephen

Tony W Wilson

Yu-Ping Wang

Abstract

I. Introduction

1). Latent Coupling Relationships within Spatio-Temporal Data:

2). The Explainability of the Model:

3). Brain dFC Analysis:

1). Constructing a Unified Framework of Spatio-Temporal Coupling Learning:

2). Building an Explainable Learning Model:

3). Validating and Applying the framework to Brain dFC Analysis:

II. STCUF-EX and its applications to the Analysis of Brain dFC

A. STCUF-EX

1). Timing Analysis (TA):

Fig. 1.

2). Node Representation Module (NR):

3). Connection Representation Module (CR):

B. STCUF-EX for Brain dFC Analysis

1). Brain Network Definition and Data Initialization:

2). Time Series Analysis-Based NR Module:

3). Time Series Analysis-Based CR Module:

III. Experimental Results and Analysis

A. Data Acquisition and Preprocessing

TABLE I.

Fig. 2.

B. Experimental Results

1). Parameter setting:

2). Significant difference connection selection:

3). Model performance analysis:

TABLE II.

TABLE III.

TABLE IV.

C. DFC Analysis

Fig. 3.

Fig. 4.

Fig. 5.

D. Differential Brain FC Analysis

Fig. 6.

IV. DISCUSSION

V. Conclusion

Acknowledgments

Contributor Information

References

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases

1). Timing Analysis ( $TA$ ):