This study utilized the Motor Imagery GigaDB database [29] from the Giga Science Database and the SHU dataset [26] collected by Shanghai University. Both databases contain extensive motor imagery and EEG data, providing a robust foundation for this study.

To ensure adequate data volume and representativeness, we selected the data of 45 subjects for our experiments from two datasets: 20 from the GigaDB and 25 from the SHU dataset. The data first underwent independent component analysis (ICA) filtering within the 8–30 Hz range to eliminate artifacts. As shown in Fig. 1, we then applied a sliding time window technique, with steps of 0.4 seconds and a width of 2 seconds, for temporal segmentation of the EEG signals. To evaluate the effect of the sliding window approach on the dynamics of EEG signals, we conducted an ablative study to identify the optimal window length and step size. After several experiments and considering the subjects’ mental focus, the duration of the EEG signal for a single trial was set at 2 seconds. Ultimately, considering computational complexity, dynamic variability, and dataset balance, we chose a sliding window step size of 0.4 second for further experiments.

Fig. 1.

Sliding time window segmentation: 0.4-sec step, 2-sec width.

To capture brain network characteristics more effectively from EEG data, we selected features apt for depicting the state of brain networks: Pearson’s correlation coefficient, coherence, and phase-locking value. These metrics provide complementary measures of linear correlation, frequency correlation, and phase synchronization, respectively. Thus, the features derived from Pearson correlation coefficient, coherence, and phase locking value offer a holistic representation of the brain network’s connectivity state.

Considering the complexity of EEG signal characteristics in brain networks, we integrate the previously mentioned metrics, PCC, COH, and PLV, along with sliding time windows to construct a 3D brain network feature matrix, referred to as temporal-frequency-phase feature (TFPF). Fig. 2 illustrates the process of decoding TFPF features using the 3D-CNN-GAN model.

Fig. 2.

Schematic diagram of the 3D-CNN-GAN for decoding TFPF features. CNN, convolutional neural networks; GAN, generative adversarial networks; TFPF, temporal-frequency-phase feature; PLV, phase locking value; ICA, independent component analysis; PCC, pearson correlation coefficient; COH, coherence; EEG, electroencephalogram.

The sequential procedure of temporal-frequency-phase feature generation is as follows: After preprocessing, the EEG signals are analyzed to extract their temporal, spectral, and phase characteristics using PCC, COH, and PLV, respectively, forming the TFPF set. Concurrently, GANs are utilized to generate synthetic EEG signals. These GAN-generated signals undergo the same process of feature extraction (PCC, COH, and PLV) to form additional TFPF. Fig. 3 depicts the formation of TFPF and the resulting voxel segments.

Fig. 3.

Process of forming TFPF. S represents the input set. Each voxel (V₁, …, V_N) is derived from EEG segments (Seg1, …, SegN). S’ represents the GAN-generated voxels. EEG, electroencephalogram.

Two types of TFPF voxels are produced: one voxel type (denoted as V) consists of EEG signals segmented by sliding time windows from the original input set (S), and the other voxel type (denoted as V’) is generated by the GAN (S’). Algorithm 1 outlines the detailed steps involved in the feature generation process:

This above Fig. 4 introduces a groundbreaking method for generating TFPF representations of EEG data and employing GANs to create synthetic TFPF representations. It involves calculating three types of features: PCC, COH, and PLV for each EEG segment. These features are stacked to form a unified feature vector for each segment. The REAL-TFPF representation is created by concatenating these feature vectors across all segments. Then, GANs are used to generate the FAKE-TFPF representation from the REAL-TFPF representation.

Fig. 4.

Generation of TFPF representation Algorithm. PCC, pearson correlation coefficient; COH, coherence; PLV, phase locking value.

The combined use of PCC, COH, and PLV metrics offers a powerful tool for exploring the brain’s connectivity and dynamics. This comprehensive approach not only facilitates the effective identification of EEG signal patterns but also aids in extracting vital insights into cognitive functions and underlying neural activities.

Employing a 3D-CNN for extracting multi-domain features represents a state-of-the-art method that amalgamates temporal, frequency, and phase data. The application of 3D-CNNs allows for the simultaneous capture of complex dynamics across these three domains, simplifying the feature extraction process and mitigating the potential reduction in accuracy associated with separate feature extraction techniques. Table 2 provides the implementation details of the 3D-CNN.

Upon acquiring three-dimensional spectral features, the limitation of conventional 2D CNNs in fully preserving the intricate details of EEG signal information—spanning temporal, frequency, and phase domains—becomes evident. This shortfall overlooks the critical interdependence among the three dimensions. Consequently, this study adopts 3D CNNs to ensure the retention of complex, interrelated patterns essential for decoding cognitive states. The superior capability of 3D CNNs in feature extraction significantly enhances the model’s efficacy in detecting and categorizing brain activities related to motor imagery. This improvement leads to higher accuracy rates and more dependable real-time responses in BCI systems.

GANs have emerged as a significant technological advancement within the realm of BCIs, demonstrating substantial promise for a variety of applications, including data augmentation [38], signal denoising and decoding [39], neural rehabilitation, and brain-to-brain communication [4]. The key aspect of GANs is their objective function formula, as outlined below (Eqn. 6):

(6)$\underset{G}{\min }\underset{D}{\max }V(D,G)=E_{x\sim pdata}(x)[\log (D(x))]$ $+E_{z\sim p_z(z)}[\log (1-D(G(z)))]$

The GAN objective function is divided into two primary parts:

Discrimination of Real Data:

E_{x∼p⁢d⁢a⁢t⁢a}(x)[$\log$ (D(x))] represents the expected value of the discriminator’s (D) accuracy in identifying samples (x) from the real data distribution pdata as real.

Discrimination of Generated Data:

E_z∼pz⁢(z)[$\log$ (1–D(G(z)))] denotes the expected value of the discriminator’s (D) accuracy in classifying samples G(z) produced by the generator (G) as fake. In this context, z represents a noise vector sourced from a specific prior distribution P_z(z).

The architecture of GANs, characterized by its innovative generator-discriminator design, is adept at learning data distributions. This facilitates advancements in the creation of synthetic data and adversarial training methods. The intricate architecture of GANs, as applied to the GigaDB and SHU datasets, is detailed in Tables 3a,3b,4a,4b.

The integration of Generative Adversarial Networks with 3D-CNN-GAN creates a formidable strategy for dataset augmentation through the generation of an enriched set of TFPF. The GAN, recognized for its proficiency in data augmentation, is adept at creating an enrich set of TFPF that accurately replicate the intricate properties of original EEG signals. This capability significantly enhances the volume and quality of training data, which in turn improves the motor imagery decoder’s ability to generalize and perform classifications. By merging GAN with 3D-CNN, the combined framework exploits the comprehensive analytical power of three-dimensional convolution, capturing a more detailed representation of EEG signal dynamics across various domains. The strategic emphasis on the 3D-CNN-GAN architecture not only extends the dataset with quality features but also strengthens the classification mechanism. This approach offers a sophisticated method for decoding the complex patterns present in EEG data.

Within the realm of EEG signal classification, it has been observed that shallow network architectures often outperform deeper networks, a trend that contrasts with observations in other tasks. In line with this finding, we have meticulously designed a neural network with just two convolutional layers, as shown in Fig. 2d. The initial convolutional layer features a kernel voxel size of 3 $\times$ 3 $\times$ 3 and a feature map dimension of 50. The kernel size for the subsequent convolutional layer is determined by the formula (C – 3 + 1) $\times$ (C – 3 + 1) $\times$ (N – 3 + 1), as indicated in Fig. 2d, where C represents the input dimension on one axis, and N denotes another input dimension. The feature map dimension for this layer is set at 100. These parameters were carefully chosen to enhance EEG data feature extraction, allowing the network to identify complex neurophysiological patterns effectively. During the training phase, our 3D-CNN employs ReLU activation functions over 100 epochs. We used the Adam optimization algorithm to reduce the cost function, conducting experiments with a batch size of 16 and a learning rate of 0.0001.

To validate our model’s stability and accuracy, for both GigaDB and SHU datasets in a within-session scenario, we divided the independent data segments into two equal parts; 50% were enhanced using a GAN network [40], resulting in an augmented set of twofold independent data segments for training. The remaining 50% independent data segments were used for testing. Hence, for the 3D-CNN-GAN, under the within-session scenario, the training data from both GigaDB and SHU datasets accounts for 2/3 of the total dataset, while the testing data accounts for 1/3 of the total dataset. For the cross-session scenario, each of the five sessions per subject is split into two equal parts due to the sliding time window strategy. We enhanced the independent data segments from the four sessions not involved in testing using GAN. Consequently, for the 3D-CNN-GAN in a cross-session scenario within the SHU dataset, each experiment’s training data accounts for 8/9 of the total dataset, with test independent data segments comprising 1/9. Therefore, this approach ensures our model’s robustness and dependability, as well as the relevance of our test data, thereby affirming the model’s credibility.

Our comprehensive brain network analysis meticulously examined the complex interconnectivity and nuanced information flow within the brain. This examination helped uncover interactions among various brain regions, elucidating the pathways of information transfer and the collaborative operations of neural functions. This contributes to our understanding of the foundational processes that govern cerebral functionality and cognitive dynamics. Fig. 5 illustrates the brain network analysis process, beginning with EEG signal processing to extract features, followed by calculating the connectivity between different brain regions, and concluding with the analysis of the differences between real and fake brain connectivity.

Fig. 5.

Schematic diagram of brain network analysis. (a) Clean EEG signal was obtained. (b) Multi-domain feature representation for the connectivity matrix. (c) REAL & FAKE Connectivity Assessment. (d) REAL & FAKE connectivity analysis using significant connections based on PCC, COH, and PLV. PCC, pearson correlation coefficient; COH, coherence; PLV, phase locking value; L, left hand; R, right hand.

To assess the efficacy of the proposed 3D-CNN and 3D-CNN-GAN models in handling EEG session variability, we conducted classification experiments across multiple sessions using the SHU and GigaDB datasets, as shown in Figs. 6,7.

Fig. 6.

The comparison of average within-session classification accuracies between 3D-CNN and 3D-CNN-GAN across the two datasets, detailing the performance for each session. NS indicates no significant difference.

Fig. 7.

The comparison of average cross-session classification accuracies between 3D-CNN and 3D-CNN-GAN within the SHU dataset, detailed for each session. **** indicates p 0.0001.

In the SHU dataset, the 3D-CNN-GAN model shows a notable improvement in classification accuracy, with within-session improvements of up to 3.99% and cross-session improvements of up to 4.98%. However, in the GigaDB dataset, the improvement with the 3D-CNN-GAN model is not as pronounced, with the 3D-CNN and 3D-CNN-GAN models achieving accuracies of 76.59% and 77.03%, respectively. The marked improvement observed in the SHU dataset can be attributed to the GAN’s ability to enhance the learning of brain network connectivity features and effectively eliminate irrelevant connections, likely due to potential external interferences encountered during EEG data collection in the SHU dataset.

To assess the 3D-CNN and 3D-CNN-GAN models’ ability to manage EEG variability across individuals, classification experiments were conducted for each subject within the SHU and GigaDB datasets. Fig. 8 shows a detailed analysis of classification accuracies for individual subjects across these two datasets, employing both 3D-CNN and 3D-CNN-GAN models. In the GigaDB dataset, the average within-session classification accuracy using 3D-CNN stands at 76.49%, with a slight increase to 77.03% observed when employing the 3D-CNN-GAN model. Notably, within this average, subjects S3 and S14 reached exceptional average accuracies of 88.87% and 91%, respectively. Conversely, in the SHU dataset, the 3D-CNN model’s within-session average accuracy of 67.42% improved to 71.63% with the 3D-CNN-GAN, and for cross-session, the average accuracy rose from 58.06% with 3D-CNN to 63.04% with 3D-CNN-GAN. A closer look reveals that, despite the overall average accuracy of 71.63% for within-session and 63.04% for cross-session with the 3D-CNN-GAN model, significant exceptions were observed. For instance, within the SHU dataset, subject S6 achieved accuracies of 88.27% within-session and 80.55% cross-session, while subject S16 reached accuracies of 84.29% within-session and 78.68% cross-session. These significant improvements in the SHU dataset highlight the GAN algorithm’s effectiveness, especially in enhancing classification accuracy per subject. The uniform improvement across all subjects in the SHU dataset with the 3D-CNN-GAN model evidences the algorithm’s capability to bolster the discriminative features of EEG signals for motor imagery classification.

Fig. 8.

Comparison of within-session and cross-session classification accuracies for 3D-CNN and 3D-CNN+GAN across two datasets, detailed for each subject.

Upon examining the subject-specific experimental results shown in Fig. 8, it is evident that SHU-S6 achieved the highest accuracy, with significant improvements attributed to GAN. Consequently, we delved deeper into the analysis of the best performer’s brain network features, revealing the reasons behind its superior performance from the perspective of brain functional connectivity. This approach facilitates an understanding and observation of the neural mechanisms underlying effective classification performance in motor imagery tasks. Analyzing connectivity metrics provides valuable insights into the collaboration and cooperation among brain regions involved in motor imagery tasks.

In this research, our research team explored brain functional connectivity by extensively analyzing the brain network of SHU-S6 from the SHU dataset. Fig. 9 presents the lateral view of brain network topology, combining PCC, COH, and PLV with GAN, to depict the connectivity across different brain regions through the color and line thickness of the marked lines.

Fig. 9.

Lateral view of brain network topology.

Our investigation revealed that the connections between key brain regions, such as the frontal, parietal, and occipital lobes, intensified during motor imagery tasks. Notably, Fig. 9 contrasts the “REAL Connectivity” and “FAKE Connectivity” within the brain network, revealing that the latter often exhibited a greater number of structures and stronger connections. Specifically, enhanced connectivity was particularly noted between the prefrontal motor area, the parietal lobe’s somatosensory associated area, and the occipital lobe’s visual associated area and visual cortex.

Upon further observation, it is exciting to note the formation of “triangular structures” and “circular structures”. Specifically, there was an observed enhancement in connectivity between the prefrontal motor area, the somatosensory associated area of the parietal lobe, and the visual associated area along with the visual cortex of the occipital lobe. Further examination revealed the intriguing formation of “triangular structures” and “circular structures” within the highlighted red border regions of the frontal, parietal, and occipital lobes. These structures are pivotal characteristics of complex networks, and an increased number of such structural features within brain connectivity signifies enhanced stability in “communication” across different brain regions. This revelation holds significant implications for a deeper understanding of EEG features and their application in brain network analysis, offering new strategies to enhance classification accuracy.

Furthermore, compared with the brain network topology of REAL and FAKE Connectivity, add the top view brain network topology and the corresponding adjacency matrix. In Fig. 10, we analyzed the network structural features using data from SHU dataset subject S6. The color and thickness of the marked lines represent the connectivity strength of different brain regions from top view brain network topology, combined with the corresponding adjacency matrix, connectivity structures, and strength of “FAKE Connectivity”, which are stronger compared to “REAL Connectivity”. Thus, from the confirmed experimental insights, we found that combining PCC, COH, and PLV with GAN not only augments the volume of data available for analysis but also potentially enriches the feature content with synthesized EEG signals that exhibit similar spectral and temporal characteristics as the real data. Experimental results confirmed that this approach not only markedly improved classification accuracy within the SHU dataset but also enhanced the model’s interpretability.

Fig. 10.

Top view of brain network topology.