For clarity, the overall workflow of the proposed method is illustrated in Fig. 1. First, the EEG signals are acquired from the DEAP dataset, which adopts music videos as stimuli. The multi-channel recordings are collected using a 32-channel system. Next, the short-time Fourier transform (STFT) is applied to convert each EEG from the time domain into the frequency domain, and PSDs are extracted based on various brain rhythms. Subsequently, rhythmic-based 2D images are projected by those extracted PSDs with spatial information, providing a comprehensive representation of channel, rhythm, and temporal properties. After that, the EEG-ERnet is designed using the depthwise parallel CNN architecture, which is then employed for training and testing the rhythmic image features through 10-fold cross-validation. Finally, the most distinguishable brain rhythms associated with specific time intervals are investigated in detail for diverse emotion recognition tasks, offering subject-independent solutions in this field.

Data acquisition is the first step in EEG studies. As the primary objective is to design a subject-independent deep learning model, a publicly available dataset is a suitable choice. Therefore, the DEAP dataset, developed by Koelstra et al. [42], was evaluated. This dataset was designed to analyze emotional states, incorporating EEG recordings and comprehensive subjective assessments. Such information makes it valuable for cross-subject evaluation.

In detail, DEAP contains EEG recordings from 32 subjects (17 males and 15 females, aged 27.19 $\pm$ 4.45 years), each of whom watched 40 one-minute music videos prepared to evoke various emotions. The physiological data include 32-channel EEG signals sampled at 512 Hz, as well as peripheral signals such as electrooculogram (EOG), electromyogram (EMG), skin temperature, and respiration. Additionally, this dataset provides a preprocessed version, including downsampled signals at 128 Hz and the removal of artifacts.

Concerning the emotional scenarios, the DEAP dataset contains ratings for three fundamental dimensions: valence, arousal, and dominance, based on a 9-point scale provided by the subjects, i.e., self-assessment manikin (SAM), as illustrated in Fig. 2. Valence denotes the degree of pleasantness of an emotional state, from negative (sadness, fear) to positive (happiness, excitement). Arousal reflects the physiological activation level, from calm (relaxation, boredom) to excitement (stress, enthusiasm). Dominance refers to the sense of control over emotional experiences, ranging from submissive (fear, helplessness) to dominant (confidence, empowerment). These dimensions form a framework for describing emotional states, where specific emotions can be mapped to different regions within this space. For instance, high valence and high arousal correspond to emotions such as joy or excitement, while low valence and high arousal denote emotions like anger or fear. Meanwhile, the liking ratings are provided, offering an additional aspect to assess subjective preferences in emotion. Therefore, this work focuses on valence, arousal, dominance, and liking tasks, where the binary classification is based on a threshold of 5, following the common practice [43] with the DEAP dataset. After binarization, the class distributions for valence, arousal, dominance, and liking are found to be reasonably balanced, with high/low class ratios of approximately 54:46, 52:48, 51:49, and 53:47, respectively. While these ratios do not represent a perfect balance, they are sufficiently close to ensure fair model training without the need for class-weighting or data balancing techniques.

The initial step in feature extraction involves obtaining the PSDs, a prevalent process in EEG emotion recognition using CNN, as incorporating PSDs as features into a CNN helps the model discern valuable emotion-associated patterns [18, 32]. To this end, the STFT is applied. The EEG x(t) is divided into overlapping segments utilizing a sliding window function to reduce spectral leakage. In this work, the window function employs the Hamming window of 128 samples with 50% overlap, ensuring a balance between time and frequency resolution. The STFT is then acquired by:

(1)$${\text{STFT}}_x(t,f)={\sum }_{n=-\mathrm{\infty }}^{\mathrm{\infty }}x(n)w(n-t)e^{-j2\pi fn}$$

where w(n-t) denotes the Hamming window function centered at time t, e^{-𝑗⁢2⁢π𝑓𝑛} represents the complex exponential term of the Fourier basis functions, which performs the transformation from the time domain to the frequency f.

The squared magnitude of the STFT provides the spectrogram from which the PSD is derived. To link with the brain rhythms, the frequency bins corresponding to each range are summed. Thus, the PSD for each brain rhythm is calculated by:

(2)$$\mathrm{PSD}(t,f)=\left|{\mathrm{STFT}}_x(t,f)\right|$$

where the PSD is integrated over the predefined brain rhythms B_K $\in$ {$\delta$, $\theta$, $\alpha$, $\beta$, $\gamma$} to generate the feature:

(3)$$P_k={\int }_{f\in B_k}\mathrm{PSD}(t,f)𝑑f$$

Once the PSDs are extracted, they are organized into spectral features for each channel. To further enhance the ability to present spatial information, these spectral features are projected onto a 2D image that preserves the spatial arrangement of high-dimensional EEG channels, i.e., these features are arranged into 2D matrices $I_k\in {ℝ}^{H\times W}$. This 2D projection way, inspired by geographical mapping, maintains the geometric relationships between adjacent EEG channels [44]. Here, the 2D image is divided into a mesh of pixels, each representing a specific channel. Based on this, the spectral features acquired from the previous step are mapped onto the corresponding pixels, generating a spatially coherent brain rhythmic representation. Therefore, the 32-channel data is projected as a 9 $\times$ 9 size 2D image, and each rhythm generates its corresponding 2D image, as illustrated in Fig. 3.

Additionally, the temporal dynamics indicate the changes in emotional responses. As a result, the inputs are generated by smaller segments, each representing a specific 5-second interval, which helps analyze the EEG signals over time and removes redundant data. For each 5-second EEG data, the PSDs of various brain rhythms are extracted and projected onto a 9 $\times$ 9 image. After that, normalization is adopted to mitigate the impact of scaling differences and enhance the convergence speed during training. In this work, the min-max normalization is adopted, which rescales the rhythmic image features to a range of 0 to 1. A sample (DEAP, subject S12, $\gamma$ rhythm, 0–5 s interval) is displayed in Fig. 4. Based on the above steps, the feature extraction process involves 12 $\times$ 5 $\times$ 40 (time intervals $\times$ brain rhythms $\times$ music videos) per subject, which serves as the input to the EEG-ERnet.

Concerning EEG-ERnet, a depthwise parallel CNN architecture is applied to address the computational efficiency and accuracy requirements for EEG emotion recognition. Its kernel design phase involves a pre-test to determine parameters for the convolutional layers. The initialization method of the convolutional kernels has been evaluated through comparative analysis, where Gaussian distribution initialization demonstrates an improvement in accuracy over zero initialization, achieving a 45% enhancement (94.58% vs. 49.88%) in a preliminary investigation. Subsequently, the kernel size has been pre-tested with 2 $\times$ 2, 3 $\times$ 3, and 4 $\times$ 4 kernels, yielding accuracies of 91.01%, 92.69%, and 94.58%, respectively. Hence, the 4 $\times$ 4 kernel size is employed.

The traditional CNN structure consists of three continuous convolutional layers without max pooling layers. The convolution filter numbers are set at 64, 128, and 256, respectively, with a 1 $\times$ 1 convolution layer employed to reduce the number of output feature maps, alleviating computational pressure in the flatten layer. In the end, a fully connected layer with 1024 neurons and a softmax classifier is utilized. Nevertheless, this traditional structure is less effective in analyzing the 2D rhythmic image features, so the depthwise convolution layers are applied, which replace conventional convolution layers with depthwise ones, maintaining the same number of filters and sizes. The depthwise convolution effectively reduces computational cost by processing each input data separately, which is more suitable for 2D features than the traditional structure.

Now, the first layer utilizes a depthwise convolution layer with 16 filters, generating 64 separate feature maps. A split layer then divides these feature maps into four sub-batches. Four parallel continuous CNNs are designed to process each sub-batch independently. Each continuous CNN consists of two convolution layers without pooling layers, with filter numbers set at 32 and 64, respectively. A concatenate layer is subsequently employed to stack the feature maps, followed by a 1 $\times$ 1 convolution layer with 64 filters to reduce the number of output feature maps. Finally, a fully connected layer with 1024 neurons and a softmax function is implemented. As this work aims at binary classification tasks, the output is a high (1) or low (0) state for arousal, valence, dominance, and liking tasks. The EEG-ERnet is depicted in Fig. 5, where its parameterized details are listed in Table 1, and a pseudocode-style algorithm (Algorithm 1) is described as follows:

Algorithm 1: Parallel EEG-ERnet Algorithm

Input: Four image views {I₁, I₂, I₃, I₄}

Output: Predicted label y $\in$ {0, 1}

1. In parallel, compute each branch output.

2. Concatenate the outputs: F = Concat(F₁, F₂, F₃, F₄).

3. Apply dropout, flattening, and a fully connected layer.

4. Compute the prediction y.

Beyond kernel size and initialization, several hyperparameters are optimized through empirical evaluation using a grid search on the training folds. The learning rate is set to 1 $\times$ 10^-3, and the batch size is chosen to be 32, balancing stability and efficiency. Training is conducted for up to 100 epochs, with early stopping if no improvement is observed for 10 consecutive epochs. A dropout rate of 0.5 is applied before the fully connected layer to reduce overfitting. The Adam optimizer is selected for its adaptive learning rate capability. ReLU is utilized as the activation function after the convolutional layers. Additionally, an L2 weight decay of 1 $\times$ 10^-5 is employed on the convolutional layers to facilitate generalization. Such hyperparameter choices contribute to the stable performance of EEG-ERnet across subjects and tasks.

Finally, to ensure subject independence, all preprocessing steps, including STFT, PSD computation, and min-max normalization, are applied independently to each trial. STFT and PSD are computed on a per-trial basis, utilizing predefined frequency ranges without any data outside the trial. The min-max normalization is performed on each rhythmic-based 2D image, with the minimum and maximum values calculated from that image alone. No statistics are shared across training and testing folds, providing strict separation and eliminating any potential information leakage in the subject-independent evaluation protocol. Then, a subject-wise 10-fold cross-validation is adopted. Specifically, the DEAP dataset, including 32 subjects, is partitioned such that each fold contains data from approximately 3–4 individual subjects, which are entirely withheld for testing, while the model is trained on data from the remaining subjects. This process is repeated ten times, so that each subject serves as the test set once. The strict separation of subjects between training and testing sets prevents subject-specific feature leakage, supporting the development of generalizable models suitable for cross-subject emotion recognition scenarios. The performance is evaluated based on the average accuracy across all ten folds, which can avoid overfitting the training data. Please note that, since brain rhythms and time intervals are key aspects assessed in this work, the training and testing are based on specific rhythmic image features extracted at the same 5-second interval across 32 subjects. To clarify the sequence of operations in EEG-ERnet, a pseudocode-style algorithm (Algorithm 2) is provided that summarizes the input, processing stages, and output. It begins with 2D input images and proceeds through the multi-branch CNNs with depthwise separable convolution layers, eventually producing the final classification output.

Algorithm 2: EEG-ERnet Classification Procedure

Input: rhythmic-based 2D image $I\in {ℝ}^{H\times W\times C}$

Output: Predicted label y $\in$ {0, 1}

1. Apply depthwise separable convolutions in 4 parallel branches.

2. Perform ReLU activation, batch normalization, and max pooling on each branch.

3. Concatenate the outputs of all branches.

4. Apply dropout, flattening, and a fully connected layer.

5. Use a softmax function to obtain the final prediction.

Here, the computational complexity per branch is approximately O(K²$\mathrm{\cdot }$H$\mathrm{\cdot }$W$\mathrm{\cdot }$C+H$\mathrm{\cdot }$W$\mathrm{\cdot }$C$\mathrm{\cdot }$D), where K is the kernel size, H and W are the image dimensions, C refers to the input channel size, and D denotes the number of filters. The final classification layer adds O(N$\mathrm{\cdot }$M) operations. This architecture is more efficient than traditional CNNs due to the use of depthwise separable convolutions, making EEG-ERnet suitable for real-time or embedded emotion recognition applications.

In this work, MATLAB R2023b (The MathWorks Inc., Natick, MA, USA) was used for programming, and the random seed was set to 42. No learning rate scheduler was applied. Training was performed on an NVIDIA ray tracing eXtreme (RTX) 3090 graphics processing unit (GPU) using compute unified device architecture (CUDA) 11.6 (NVIDIA Corp., Santa Clara, CA, USA) on Ubuntu 20.04 (Canonical Ltd., London, UK), with early stopping applied after 10 epochs of no improvement. The same configuration was employed across all cross-validation folds and rhythm-interval evaluations. Extensive experiments were conducted based on four tasks: valence, arousal, dominance, and liking. Consequently, it is meaningful to identify distinguishable rhythms and appropriate 5-second intervals to recognize different emotions, offering insights into the characteristics of emotion recognition. To this end, all classification results were evaluated using the mean and standard deviation of accuracy across the 10-fold cross-validation for the valence, arousal, dominance, and liking tasks, as detailed in Tables 2,3,4,5. No inferential statistical tests were applied, as the primary goal of this work is to assess network model performance across rhythm-specific image inputs rather than test specific hypotheses. Also, please note that no demographic covariates were included due to the limited metadata available in the dataset.

Table 2 presents classification accuracies for valence, showing the 55–60 s interval where the $\alpha$ rhythm achieves the highest accuracy of 93.27 $\pm$ 3.05%. The $\alpha$ rhythm is typically associated with a relaxed yet attentive mental state, characterized by self-referential thought and introspection [45]. During the 55–60 s interval of the music video, it is plausible that most subjects have settled into a state of reflective calmness, which may enhance the neural correlates associated with valence [46]. This state could facilitate more precise differentiation between positive and negative stimuli. That means the heightened activity potentially enhances the neural representation of valence, resulting in improved classification performance.

Table 3 focuses on the arousal task and reveals that the $\beta$rhythm during the 50–55 s interval achieves the highest accuracy of 92.16 $\pm$ 2.73%. The $\beta$ rhythm is generally associated with cognitive functions and increased alertness. According to the results, heightened arousal during the 50–55 s interval of the music video is associated with improved mental performance and emotional engagement, as reflected by the $\beta$ rhythm. This finding aligns with the understanding that arousal can modulate neural activity [47]. Additionally, the $\beta$ rhythm at 50–55 seconds could indicate a response to the music video stimulus, requiring a longer duration for cognitive processing, similar to the valence.

Table 4 indicates that the $\theta$ rhythm provides the highest dominance classification accuracy of 90.56 $\pm$ 4.44% during the 55–60 s interval. The $\theta$ rhythm is known to be involved in memory, learning, and decision-making processes [48]. Its association with the perception of dominance may stem from its role in integrating sensory information and processing complex emotions [49]. The enhanced classification accuracy observed in the 55–60 s interval could be due to the $\theta$ rhythm’s involvement in binding emotional information with contextual cues, which are vital for recognizing dominance-related signals, such as music video stimuli.

Table 5 shows that the $\gamma$ rhythm offers the highest classification accuracy of 86.68 $\pm$ 5.66% in the 5–10 s interval for the liking task. The $\gamma$ rhythm is associated with higher cognitive functions, where its prominence in the early interval suggests a rapid neural processing mechanism for music video stimuli, mainly due to the brain’s quick evaluation of sensory input, leading to an immediate emotional reaction [50]. This response is helpful for social interactions, where swift identification of positive stimuli is beneficial. That may be why most subjects determine their levels of liking at the beginning of emotional stimuli.

Finally, the varying accuracies across classification tasks may be attributed to inherent emotional complexity. Emotions, such as valence and arousal, are two fundamental dimensions that are widely recognized for understanding human emotions. Dominance and liking may involve more subjective and context-dependent interpretations. In this regard, the proposed EEG-ERnet demonstrates its ability to maintain impressive performance across individuals and tasks, since its architecture incorporates depthwise parallel CNNs that can effectively analyze both local and global features of rhythmic-based 2D images. The explainability of EEG-ERnet further contributes to providing insights into the brain rhythms and time intervals that are most beneficial in emotion recognition, improving its decision-making process in various cases across subjects. It is vital for real-world applications where emotional recognition systems should be adaptable to different users and contexts.

A comprehensive comparative study was conducted to evaluate the proposed EEG-ERnet. First, in the ablation experiment, an initial baseline model consisting of three depthwise convolution layers, a 1 $\times$ 1 convolution layer, and a fully connected softmax function was compared. This initial version serves as the basis for the EEG-ERnet. The results are listed in Table 6.

Table 6 shows that the baseline model, which utilizes cascading depthwise convolution layers, suffers from a limitation in its architecture. Depthwise convolution, while computationally efficient, lacks feature integration. Hence, it cannot fully exploit the multi-dimensional nature of the data to make accurate classifications. In contrast, EEG-ERnet employs a parallel architecture that enhances feature integration, resulting in improved performance across all dimensions. Such an architectural advantage makes the EEG-ERnet more appropriate for cross-subject emotion recognition tasks.

A comprehensive overview of various EEG emotion recognition methods utilizing the DEAP dataset is offered by the comparative study presented in Table 7 (Ref. [14, 18, 29, 34, 51, 52, 53, 54, 55]). Different approaches have been assessed, including traditional machine learning algorithms such as k-NN, SVM, and RF, as well as more advanced deep learning techniques like CNN and its variants. Mahmoud et al. [18] employed a 2D-CNN with PSD features, achieving the highest valence and arousal recognition accuracy of 94.23% and 93.78%, respectively. However, their method did not cover all dimensions. The proposed EEG-ERnet has demonstrated impressive performance across all dimensions, achieving accuracies of 93.27% for valence, 92.16% for arousal, 90.56% for dominance, and 86.68% for liking. It is advantageous over previous works, particularly in terms of stable performance across subjects and emotional factors, which are key concerns.

Compared to methods that rely on temporal snapshots or spectrogram images, the EEG-ERnet enables the analysis of spatial and temporal information from rhythmic image features. It exhibits several advantages in EEG emotion recognition. First, by utilizing depthwise convolution layers, the model analyzes the spatial information of EEG signals more effectively than traditional CNNs. This architecture reduces computational costs and enhances the ability to identify spatial patterns associated with specific emotions. Second, the parallel processing of sub-batches offers more comprehensive feature integration, allowing the model to handle high-dimensional and complex data. As demonstrated in the experiments, such design choices collectively contribute to superior performance across four dimensions. Additionally, the results provide valuable insights into the characteristics of EEG emotion recognition, as they identify specific brain rhythms and time intervals associated with recognizing emotions. Such findings align with the known roles of these rhythms in emotional processing. Thus, the model identifies the key factors relevant to particular dimensions. Overall, the accuracies acquired in four dimensions demonstrate that it is well-suited to recognize the complex patterns in EEG signals associated with different emotions. It can be said that the EEG-ERnet provides an impressive solution that outperforms existing methods by addressing the limitations of incomplete dimensions in a subject-independent manner with only 5-second data sources.

First, this work focuses on the five brain rhythms due to their strong neurophysiological grounding in the context of emotion recognition. While emerging techniques such as EMD can extract non-standard or adaptive frequency components, the proposed method prioritizes choice based on explainability and computational feasibility. Nevertheless, it is recognized that the potential of such methods, including high-frequency bands beyond 60 Hz, lies in uncovering additional emotion-relevant information. Future work will explore the integration of EMD-derived IMFs into the 2D image framework to enhance the discriminative ability for multiple-level classification tasks.

Second, the choice of a 5-second interval in this work is due to the balance between temporal resolution and spectral stability. This duration has also been used in emotion recognition and aligns with previous works [3, 4] on the DEAP dataset. A shorter interval, like 2 seconds, may offer finer temporal granularity. Still, they could suffer from reduced frequency resolution, particularly for low-frequency bands. In contrast, a longer interval, such as 10 seconds, may average out dynamic changes and reduce the number of training samples. Although this work adopts a fixed non-overlapping 5-second interval, future work will investigate the effects of different segmentations, including overlapping and multi-scale windows, to improve real-time responsiveness and model performance.

Third, min-max normalization is applied to each rhythmic image to rescale PSD values to the [0, 1] range, which provides a consistent input scale across samples and reduces the impact of extreme values. While this method preserves the relative topographic and spectral structure of each input, it does not standardize inter-subject statistics. Therefore, to mitigate inter-subject variability, a subject-independent cross-validation involves training and testing on entirely different sets of individuals. Although no explicit domain adaptation is adopted, it promotes cross-subject generalization. Future work will incorporate advanced inter-subject normalization or domain-adaptive learning techniques, such as Riemannian alignment, statistical matching, or adversarial adaptation, to further enhance model robustness in highly diverse populations.

Next, the DEAP dataset assessed in this work includes EEG recordings from 32 subjects, spanning a reasonable demographic range. But it does not represent specific clinical populations. Also, it is relatively small compared to datasets in other EEG-based studies, which may constrain the model’s generalizability to broader or clinical populations. Meanwhile, in the DEAP dataset, the emotional responses may be influenced by cultural or psychological factors not accounted for, and no inferential statistical tests or covariate analyses are performed based on the results obtained, which limits the investigation of group differences, a limitation regarding personalized affective modeling. Future work will involve larger, more diverse studies and the introduction of covariate-aware modeling to enhance personalization.

Furthermore, EEG signals in the DEAP dataset may reflect overlapping cognitive phenomena beyond core emotional responses, including subjective perceptions like luck, expectation, or decision uncertainty. These non-emotional components can confound emotion recognition. To address this issue, the proposed method mitigates it by using band-specific rhythmic representations and parallel CNN branches, which help isolate emotion-relevant frequency dynamics. Meanwhile, the use of subject-independent training emphasizes generalizable emotion-related features while suppressing subject-specific cognitive noise. Future work will incorporate explicit component separation techniques such as adversarial learning to disentangle emotional signals from co-occurring cognitive influences.

Finally, this work identifies the brain rhythm and temporal interval combinations that are most informative for each emotional dimension. To this end, the experiments have thoroughly evaluated the model’s performance across 60 rhythm-interval configurations in a controlled and consistent cross-validation setting. All evaluations are performed using subject-independent cross-validation, guaranteeing that performance is not inflated due to subject-specific overfitting. Therefore, the results add a layer of generalization to mitigate risk from testing multiple configurations. Such an approach is suitable for portable emotion-aware devices, as it utilizes fewer data sources with only 5-second data. In the future, a multi-configuration approach will be considered to enhance performance across various cases. Trivial baselines and nested cross-subjects will also be incorporated into framework-related applications, such as those for depression detection, to improve the model’s advantage.