, Pengyan Zhou 2, Jinhan Liu 1, Yixuan Guan 1, Jiacheng Qian 3, Jiayi Bao 1,*
1 School of Aeronautic Science and Engineering, Beihang University, 100191 Beijing, China
2 School of Biological Science and Medical Engineering, Beihang University, 100191 Beijing, China
3 College of Mathematics and Physics, Beijing University of Chemical Technology, 100029 Beijing, China
Abstract
Pilots often experience mental fatigue during task performance, accompanied by fluctuations in positive (e.g., joy) and negative (e.g., tension) emotions. Both mental fatigue and emotional changes significantly contribute to aviation accidents, yet few studies have considered their interplay. This study had three primary objectives. First, it examined the changes in positive and negative emotions following mental fatigue. Second, it investigated how these emotions influence the recovery from fatigue. Finally, it developed a comprehensive evaluation model integrating mental fatigue and emotional states.
Two task sets were created using the visual search paradigm, incorporating simulated flight tasks with positive and negative emotional stimuli. Data were collected from 30 participants using electroencephalogram (EEG), eye-tracking, electrocardiogram (ECG), and behavioral performance metrics.
Participants showed mental fatigue after the simulated flight task, with reduced arousal for both positive and negative emotions; positive images had stronger effects. ERP showed decreased N1, P3, and LPP amplitudes. A Support Vector Machine (SVM) classifier achieved over 93% accuracy for fatigue but about 70% for emotion recognition.
The task effectively induced fatigue and indicated that positive stimuli may aid recovery. Multimodal features support accurate fatigue detection, though emotion classification needs improvement
No: ChiCTR2500104961. https://www.chictr.org.cn/showproj.html?proj=267844.
Keywords
- simulated flight
- mental fatigue
- emotional arousal
- SVM
Pilots face dual challenges of mental fatigue and emotional fluctuations during mission execution [1, 2, 3]. Mental fatigue is one of the most critical factors leading to a decline in pilot performance, posing a significant threat to aviation safety [1, 4, 5]. It manifests across various dimensions, including cognitive, psychological, emotional, and behavioral [6]. Psychologically, fatigue appears as tiredness, headaches, and palpitations, while behaviorally, it is characterized by distractibility and drowsiness [7, 8, 9]. Emotionally, it often leads to anxiety, reduced self-confidence, and irritability [10]. When pilots experience mental fatigue, their levels of alertness and attention drop significantly [11, 12]. For example, 72% of Army pilots flying under sleep-deprived conditions are at risk of falling asleep during flight [13]. A survey of 162 experienced U.S. Air Force pilots and navigators revealed that fatigue-induced performance degradation is common across cockpits, often resulting in reduced situational awareness and delayed reaction times [14].
Meanwhile, emotions are also closely related to aviation safety [15, 16]. Positive emotions, such as happiness and alertness, correlate with optimal performance, whereas negative emotions, such as anxiety, often emerge during suboptimal performance [17]. Emotionally demanding tasks, particularly those involving complex working memory, have been shown to impair executive functions under emotional stress [18]. During prolonged cognitive tasks, individuals often report increased fatigue and negative emotions while experiencing a marked decline in attention and positive emotions [19]. In simulated unsafe landing scenarios, negative emotional outcomes associated with go-around decisions have been found to temporarily impair decision-making processes, increasing aviation risks [20]. Beyond flight tasks, pilots’ emotional changes driven by social pressures also impact flight safety. Prolonged exposure to emotionally stressful work environments can lead to negative workplace behaviors and diminished performance [21]. In a simulated flight experiment where task difficulty was a secondary factor, it was found that social stressors and cognitive workload manipulation influenced emotional responses. Specifically, social pressure sources triggered emotional reactions that enhanced secondary task engagement and performance [22].
Fatigue and emotional states in pilots are typically assessed through induced scenarios in simulated or real-flight environments, followed by analyzing performance and physiological data. Measurement tools include subjective questionnaires, physiological monitoring, and performance evaluations [23]. Among these, physiological metrics are particularly valued for their objectivity and sensitivity. Methods include collecting and analyzing electroencephalogram (EEG), electrocardiogram (ECG), eye-tracking, electromyogram (EMG), and blood oxygen saturation data. In EEG-based measurements, event-related potential (ERP) is commonly used to assess fatigue and emotional responses. Key ERP components include N170, P300, and late positive potential (LPP). The N170 component, peaking between 130–200 ms, is primarily associated with emotional processing and is most prominent in the right hemisphere [24]. The P300 component peaks around 300–600 ms during emotional picture tasks, with maximal activity in the central-parietal region, reflecting both emotional and non-emotional processes [25]. The LPP component, starting around 400 ms, is more sensitive to arousal levels and emotional regulation, persisting throughout stimulus presentation [26]. For mental fatigue, researchers focus on ERP components such as P1, N1, P2, N2, and P300, as changes in their latency and amplitude can reflect alterations in attention post-fatigue [27]. EEG frequency bands are also significant, with beta activity positively correlated with alertness and alpha/theta activity negatively correlated with it, serving as indicators of mental fatigue [28]. Similarly, heart rate (HR) and heart rate variability (HRV) metrics are commonly used as a physiological marker of autonomic nervous system function, particularly reflecting the balance between the parasympathetic and sympathetic branches. The high-frequency component of HRV is associated with parasympathetic activity, while the low-frequency component is considered a marker of both sympathetic and parasympathetic influences [29]. Time-on-task effects on HRV are linked to a decrease in parasympathetic activity, with time-domain indices providing more reliable monitoring of autonomic changes during mental fatigue induction. In contrast, frequency-domain indices are more closely related to psychological symptoms of mental fatigue [30]. Mental fatigue may involve two opposing processes: task disengagement and compensatory effort, both of which are influenced by the sympathetic and parasympathetic nervous systems. Therefore, a significant study has explored the relationship between HRV and mental fatigue [31]. Meanwhile, eye-tracking features like blink frequency and pupil diameter are widely used for fatigue detection [32, 33].
Recent studies leverage machine learning to classify fatigue and emotional states with high accuracy. For instance, Chen et al. [34] used graph neural networks to classify emotional states in simulated driving scenarios with 75.26% accuracy. Lee et al. [35] applied a CNN-based multi-feature block algorithm to classify pilots’ mental states (normal, fatigued, workload-induced, and distracted) with high precision. Yu et al. [36] combined eye-tracking and EEG data to build a convolutional neural network-long short-term memory model, accurately identifying pilots’ fatigue levels. Similarly, Qin et al. [37] achieved 91.8% accuracy in mental fatigue detection using ECG and eye-tracking data.
Despite these advancements, the interaction between fatigue and emotions remains poorly understood. Specifically, the reciprocal effects of fatigue on emotional arousal and how positive or negative emotions either amplify or mitigate fatigue are unclear. For instance, Ghanbari et al. [38] used emotional imagery as a stimulus to evaluate brain arousal post-fatigue but found no significant results. Conversely, Fan et al. [39] demonstrated that mindfulness meditation could counteract the negative associations between fatigue and emotional neural activation by maintaining LPP amplitudes during emotional stimuli.
In this study, experiments were designed to induce both positive and negative emotions as well as mental fatigue in simulated flight tasks, with the scenarios based on a visual search paradigm. The data collected during the experiments included EEG, ECG, eye-tracking metrics, work performance, and subjective questionnaires. The study aimed to address three key objectives: (1) Does the simulated flight task successfully induce mental fatigue, and what physiological indicators exhibit significant changes? (2) How do positive and negative emotions change after mental fatigue, and what is the arousal effect of these emotions on mitigating mental fatigue? (3) Considering the coupling between mental fatigue and emotional changes during flight tasks, is it possible to classify these two states effectively?
To minimize the influence of factors such as educational background and age, the study recruited 30 healthy male volunteers, all right-handed, aged 22 to 26, from the postgraduate student population of Beihang University. Before the experiment, participants were instructed to maintain adequate sleep and a normal diet for two days and to abstain from consuming stimulants such as strong tea or coffee for 24 hours. The sample size was close to the optimal sample size calculated in G*Power for ANOVA, f = 0.25, a = 0.05, 1-beta = 0.80, Number of groups = 1, Number of measurements = 4, resulting in N = 24.
During the testing sessions, participants were required to stay focused and minimize actions like speaking, swallowing, chewing, or making unnecessary movements. Upon completing the experiment, volunteers were compensated for their participation. This study was performed according to the Declaration of Helsinki and was approved by the Biological and Medical Ethics Committee of Beihang University (approval: BM20250004). All adult participants provided written informed consent to participate in this study.
The experiment employed two paradigms: an emotional image recognition task (emotional experimental paradigm) and a visual search task (visual search paradigm). For the emotional image recognition task, 120 pictures (40 positive, 40 negative, and 40 neutral) were selected from the Chinese Affective Picture System (CAPS) as emotional stimuli. The CAPS database demonstrates high internal consistency (Cronbach’s
For the visual search task, a total of 40 dashboard images were used as stimuli, including 20 target images and 20 non-target images. Each image was arranged in a 3
The experiment comprised two sub-experiments: a positive emotion condition and a negative emotion condition, both of which were completed by each participant. A 20-minute break was provided between the two sub-experiments to allow participants to recover. Each sub-experiment included three emotional task blocks and five visual search task blocks. The visual search tasks were designed to simulate cockpit instrument monitoring by pilots, while the emotional image tasks intended to elicit either positive or negative emotional states.
In the positive emotion condition, participants first completed a positive emotional image task, followed by five consecutive visual search tasks, and concluded with another positive emotional image task. The negative emotion condition adopted an identical structure. The visual search tasks positioned between the two emotional tasks were collectively referred to as fatigue-inducing tasks, aimed at eliciting mental fatigue in participants. Additionally, participants were required to complete a visual analogue scale (VAS) before and after the experiment to subjectively assess their mental state. The VAS included four dimensions: alertness, anxiety, vitality, and sleepy.
In the positive emotional task, participants were presented with a random sequence of images drawn from two categories: positive and neutral, each comprising 50% of the stimuli. Each image was displayed for 1000 ms, followed by an inter-stimulus interval of 1000 ms. The total task duration was 160 s. The negative emotional task followed the same design, but featured negative and neutral images instead. All parameters (image count, duration, and interval) were consistent across both conditions. Participants were instructed to click the left mouse button when viewing a positive or negative image and the right mouse button when viewing a neutral image. Each emotional block lasted approximately 160 s. Behavioral performance in each block was assessed via reaction time (RT) and accuracy (Acc). The total duration of the experiment was approximately 3 hours. The overall experimental procedure is illustrated in Fig. 1.
Fig. 1. Overall experimental design. VAS, visual analogue scale.
In this experiment, EEG, ECG, eye-tracking, and behavioral data were collected from participants. The EEG signals were recorded using the NeuroScan 40 system (Compumedics NeuroScan, Charlotte, NC, USA) with a sampling rate of 1000 Hz. ECG data were acquired using the MP150 Data Acquisition System (BIOPAC system, Inc., Goleta, CA, USA), also with a sampling rate of 1000 Hz. Eye-tracking data were collected using the desktop eye-tracker (SensoMotoric Instruments, Germany). The specific experimental setup is shown in Fig. 2.
Fig. 2. The experimental environment.
The data preprocessing of EEG signals was analyzed using the EEGLAB toolbox (https://sccn.ucsd.edu/eeglab/index.php) in the MATLAB 2023b (MathWorks, Inc., Torrance, CA, USA) environment. Four electrooculogram (EOG) channels (‘HEOL’, ‘HEOR’, ‘VEOU’, ‘VEOL’) were excluded from acquisition due to irrelevance to the analysis objectives. Additionally, electrodes ‘FT9’, ‘FT10’, ‘PO1’, and ‘PO2’ were removed post-acquisition due to poor signal quality, resulting in 32 retained channels. All preprocessing steps were performed using the EEGLAB toolbox running under MATLAB 2023b. Initially, EEG datasets were re-referenced to the average of the mastoid electrodes, ‘A1’ and ‘A2’, to establish a common reference. Subsequently, the continuous EEG data were filtered using a zero-phase Butterworth bandpass filter with cutoff frequencies of 1 Hz and 40 Hz to remove slow drifts and high-frequency noise. A notch filter centered at 50 Hz was also applied to eliminate line noise contamination. Following filtering, the continuous data were segmented into 1.2 s epochs (–0.2 to 1.0 s relative to stimulus onset), with the pre-stimulus interval (–0.2–0 s) used for baseline correction via mean voltage subtractions. To address ocular and other non-brain artifacts, independent component analysis (ICA) was employed. An experienced researcher visually inspected the ICA decomposition results, and artifactual components (typically 1–5 components per EEG dataset) were manually identified and removed.
An ERP analysis was performed on the EEG data collected from the frontal (FZ), central (CZ), and parietal (PZ) electrodes during emotional tasks for both the positive and negative groups. These locations overlie key cortical areas, including the prefrontal cortex, central parietal regions, and parieto-occipital cortex. Consequently, they are considered crucial recording sites in ERP studies for examining broadly distributed cognitive processes such as attention, inhibitory control, and working memory [39, 41, 42]. FZ: Positioned over the prefrontal cortex, primarily associated with emotional regulation and task control. CZ: Located over the central cortex, involved in sensory processing. PZ: Situated over the parietal-occipital region, playing a crucial role in visual processing. Specific time windows for key ERP components were defined to analyze neural activity during the tasks, including the N1 (100–150 ms), P2 (150–250 ms), N2 (200–350 ms), P3 (300–500 ms), and LPP (500–1000 ms). These intervals were carefully selected to capture neural responses critical for attention allocation, cognitive processing, and sustained emotional evaluation, aligning with the expected timing of these ERP components in the context of emotional and fatigue-related tasks.
Wavelet packet energy analysis was employed to evaluate EEG data collected before and after the fatigue tasks. For each experimental condition, we extracted the relative energy features from all channels. Specifically, after performing wavelet decomposition to extract the delta (
ECG signals were processed with the NeuroKit2 toolbox (https://neuropsychology.github.io/NeuroKit) in Python 3.9.20 (Python Software Foundation, Wilmington, DE, USA), involving preprocessing that included applying a high-pass filter at 0.5 Hz and notch filtering to eliminate 50 Hz power line interference. For feature extraction, a 60-second sliding window with a 30-second step size was used to derive HR and HRV. HRV time-domain metrics included mean normal-to-normal interval (NN), standard deviation of NN intervals (SDNN), root mean square of successive differences in NN intervals (RMSSD), percentage of NN intervals differing by more than 50 ms (pNN50), and triangular interpolation of NN interval distribution (TINN), while HRV frequency-domain metrics comprised total power (TP) and LF/HF (the low-frequency (LF, 0.04–0.15 Hz) to high-frequency (HF, 0.15–0.4 Hz) power ratio). The specific formulas for each indicator are as follows:
The eye movement data analysis was performed using the BeGaze software (SensoMotoric Instruments, Warthestraße, Teltow, Germany) included with the device for processing. The features collected from the eye movement data included pupil diameter, blink duration, blink frequency, saccade amplitude, and saccade velocity. In the positive emotion task, eye movement data from two emotion stimulus blocks (before and after the fatigue task) were used for analysis, with the data duration set at 160 s. The extracted feature data were averaged for each dimension across all participants using Python 3.9.
To facilitate statistical analysis, we operationalized two categorical variables: “Positive Group” and “Negative Group”, which distinguished participants performing positive vs. negative emotion experimental conditions. Each emotional condition adopted a 2
For subjective scales, ECG features, and eye-tracking data, paired t-tests were used to compare the differences in each feature before and after the fatigue task. The significance level was set at p
For ERP data, the image-evoked components N1, P2, N2, P3, and LPP were analyzed using a 2
For behavioral data, reaction time was defined as the time elapsed between stimulus onset and correct keypress. Acc was defined as the proportion of correctly identified stimuli. RT and Acc data were subjected to the same 2
Eye movement, ECG, and subjective questionnaire data were used to verify that the designed simulated flight experiment successfully induced mental fatigue in the participants. Meanwhile, EEG and behavioral data simultaneously characterized the mutual influence between mental fatigue and emotion.
As illustrated in Fig. 3, participants’ alertness significantly decreased after completing the task, while anxiety levels significantly increased. Similarly, vigor showed a significant decline, and sleepy markedly increased.
Fig. 3. Subjective scale scores before and after the task. The vertical error bars represent the standard error, indicating the dispersion or inter-individual variability of the data for each condition. The asterisk (*) denotes statistical significance at the p
We analyzed several indicators, including pupil diameter, blink frequency, blink duration, saccade amplitude, and saccade velocity before and after the fatigue task, as shown in Table 1. After experiencing the fatigue task, participants exhibited a decrease in pupil diameter (Before: 5.09
| Pupil diameter (mm) | Blink frequency (Hz) | Blink duration (ms) | Saccade amplitude (°) | Saccadevelocity (°/s) | |
| Before task | 5.09 | 0.24 | 234.11 | 16.72 | 139.96 |
| After task | 4.51 | 0.19 | 261.96 | 11.86 | 140.24 |
| p-value | 0.0004*** | 0.72 | 0.07 | 0.12 | 0.23 |
*** means p
Paired t-test results for before and after task data are presented in Table 2. After completing the fatigue task, HR significantly decreased (Before: 75.1
| HR (bpm) | NN (ms) | SDNN (ms) | TP | pNN50 | LF/HF | TINN (ms) | RMSSD (ms) | |
| Before task | 75.1 | 759.45 | 42.71 | 869.16 | 9.84 | 2.19 | 103.61 | 28.69 |
| After task | 72.58 | 793.08 | 51.21 | 1189.1 | 14.04 | 2.45 | 130.62 | 33.71 |
| p-value | 0.018* | 0.02* | 0.03* | 0.14 | 0.46 | 0.05* |
FR, heart rate; NN, mean normal-to-normal interval interval; SDNN, standard deviation of NN intervals; TP, total power; pNN50, percentage of NN intervals differing by more than 50 ms; LF/HF, low-frequency/high-frequency; TINN, triangular interpolation of NN interval distribution; RMSSD, root mean square of successive differences in NN intervals. * means p
The independent variables in our analysis were emotional type (positive vs. neutral, negative vs. neutral) and fatigue type (awake, fatigue), while the dependent variable was the amplitude of ERP components.
As shown in Fig. 4, emotional stimulus pictures elicited distinct N1, P2, N2, P3, and LPP components. The P2 component displayed a prominent large-amplitude peak around 200 ms, the P3 component showed high amplitude between 300–500 ms, the N1 component had a large amplitude around 150 ms, the N2 component exhibited high amplitude around 250 ms, and the LPP component demonstrated large amplitude with extended duration (600–800 ms).
Fig. 4. Positive group and negative group event-related potentials. (a) Group Positive, FZ. (b) Group Positive, CZ. (c) Group Positive, PZ. (d) Group Negative, FZ. (e) Group Negative, CZ. (f) Group Negative, PZ.
In the positive group, as shown in Fig. 4a–c, after the fatigue task, both positive and neutral stimuli showed fatigue-related changes, including a reduction in amplitude for the N1, N2, P2, P3, and LPP components. The results of the two-way ANOVA for the positive group (Table 3) showed significant effects for the State factor.
| N1 | P2 | N2 | P3 | LPP | |||||||
| F | p-value | F | p-value | F | p-value | F | p-value | F | p-value | ||
| State | FZ | 44.38 | 0.06 | 0.80 | 64.27 | 113.1 | 75.35 | ||||
| CZ | 24.37 | 0.001 | 0.97 | 46.5 | 33.02 | 115.1 | |||||
| PZ | 7.81 | 0.006** | 4.69 | 0.03* | 10.76 | 2.03 | 0.15 | 1.99 | 0.16 | ||
| Condition | FZ | 0.004 | 0.95 | 1.41 | 0.23 | 0.69 | 0.41 | 44.36 | 182.9 | ||
| CZ | 2.17 | 0.14 | 1.77 | 0.18 | 3.41 | 0.07 | 47.82 | 247.8 | |||
| PZ | 0.01 | 0.93 | 16.40 | 39.72 | 102.3 | 79.89 | |||||
| Interaction | FZ | 13.96 | 1.76 | 0.19 | 13.03 | 9.26 | 0.002** | 84.9 | |||
| CZ | 39.4 | 3.28 | 0.07 | 9.35 | 0.002** | 5.54 | 0.02* | 93.61 | |||
| PZ | 3.73 | 0.06 | 12.59 | 4.06 | 0.04* | 2.63 | 0.11 | 22.4 | |||
LPP, late positive potential. * means p
After the fatigue task, the N1 component showed significantly reduced amplitudes at FZ (p
In the negative group, as shown in Fig. 4d–f, negative stimuli reduced amplitudes across multiple ERP components compared to neutral stimuli, indicating mental fatigue following the fatigue task. After the fatigue task, differences between negative and neutral stimuli diminished across components. Moreover, fatigue had a more pronounced weakening effect on negative stimuli than on neutral stimuli. The results of the two-way ANOVA for the Negative Group (Table 4) showed significant effects for the State factor.
| N1 | P2 | N2 | P3 | LPP | |||||||
| F | p-value | F | p-value | F | p-value | F | p-value | F | p-value | ||
| State | FZ | 20.48 | 0.10 | 0.75 | 9.64 | 0.002** | 15.15 | 66.9 | |||
| CZ | 46.27 | 0.63 | 0.43 | 0.01 | 0.92 | 0.04 | 0.84 | 3.51 | 0.06 | ||
| PZ | 14.26 | 6.98 | 0.01** | 10.34 | 0.002** | 9.54 | 0.002 | 0.26 | 0.61 | ||
| Condition | FZ | 84.26 | 0.11 | 0.75 | 4.99 | 0.03* | 9.38 | 0.002** | 188.1 | ||
| CZ | 51.55 | 1.67 | 0.20 | 8.63 | 0.004** | 3.22 | 0.07 | 330.1 | |||
| PZ | 3.83 | 0.053 | 5.54 | 0.02* | 0.15 | 0.70 | 14.81 | 379.1 | |||
| Interaction | FZ | 0.88 | 0.35 | 1.04 | 0.31 | 4.70 | 0.03* | 0.32 | 0.57 | 47.66 | |
| CZ | 0.07 | 0.79 | 0.34 | 0.56 | 0.25 | 0.62 | 0.02 | 0.90 | 21.84 | ||
| PZ | 0.38 | 0.54 | 0.05 | 0.82 | 0.98 | 0.15 | 0.70 | 33.41 | |||
* means p
For the negative group, the N1 component showed significant state differences at FZ (p
When participants reach a state of mental fatigue, the arousal effects of positive and negative emotional stimuli differ. Assuming that neutral emotion stimuli images have the same impact on participants’ amplitude changes, we used the amplitude values of neutral emotion images from the visual search task in both groups as a baseline. We then compared the amplitude differences between positive, negative, and neutral emotion images, as shown in the Fig. 5.
Fig. 5. Positive group and negative group event-related potentials. (a) FZ. (b) CZ. (c) PZ.
In the Positive group, the amplitude difference is calculated by subtracting the amplitude induced by neutral emotion images from the amplitude induced by positive emotion images after mental fatigue. Similarly, in the Negative group, the amplitude difference is obtained by subtracting the amplitude induced by neutral emotion images from the amplitude induced by negative emotion images after mental fatigue.
As shown in Table 5, except for the N1 component, the amplitude changes induced by positive emotion images are larger than those induced by negative emotion images at the FZ, CZ, and PZ electrodes. After mental fatigue, participants seem to experience better fatigue awakening effects when exposed to positive emotion stimuli.
| N1 | P2 | N2 | P3 | LPP | ||
| FZ | Positive | 0.26 | 1.17 | 1.14 | 1.82 | 0.90 |
| Negative | 0.45 | –0.31 | 0.04 | 0.61 | 1.19 | |
| p-value | 0.07 | |||||
| CZ | Positive | 0.65 | 1.32 | 0.43 | 1.91 | 1.25 |
| Negative | 0.87 | –0.25 | 0.52 | 0.48 | 0.81 | |
| p-value | 0.30 | |||||
| PZ | Positive | 0.26 | 1.16 | 1.01 | 1.82 | 0.90 |
| Negative | 0.48 | –0.31 | 0.04 | 0.61 | 1.19 | |
| p-value | 0.03* |
* means p
The wavelet energy distribution for the positive and negative groups in each frequency band is shown in Fig. 6a,b, respectively. After completing the visual search tasks, the relative energy in the
Fig. 6. EEG wavelet energy of the positive group and the negative group. (a) Group Positive. (b) Group Negative.
The two-way ANOVA results are presented in Table 6. In the positive emotion group, the
| δ | θ | ||||||||
| F | p-value | F | p-value | F | p-value | F | p-value | ||
| Positive | State | 2.73 | 0.108 | 4.39 | 0.033* | 0.915 | 0.346 | 0.324 | 0.573 |
| Condition | 2.391 | 0.132 | 20.311 | 16.546 | 1.618 | 0.213 | |||
| Interaction | 0.44 | 0.512 | 8.671 | 0.006** | 9.058 | 0.005** | 0.163 | 0.689 | |
| Negative | State | 0.657 | 0.006** | 0.082 | 0.777 | 0.276 | 0.603 | 0.841 | 0.001*** |
| Condition | 0.85 | 0.772 | 30.698 | 11.682 | 0.002** | 3.85 | 0.058 | ||
| Interaction | 0.903 | 8.45 | 0.006* | 8.124 | 0.007** | 0.260 | 0.037* | ||
* means p
In the negative emotion group, the
Performance data on RT and Acc for the positive group are illustrated in Fig. 7a,b, respectively. After the visual search task, participants exhibited increased mean reaction times and decreased mean accuracy across all conditions, indicating signs of fatigue.
Fig. 7. Reaction time and accuracy of the positive group and the negative group. (a) Group Positive. (b) Group Negative. PP, the positive stimuli of the positive group; PN, the neutral stimuli of the positive group; NP, the negative stimuli of the negative group; NN, the neutral stimuli of the negative group.
Two-way ANOVA results (Table 7) reveal that reaction times significantly increased following the visual search task, reflecting a clear decline in performance. Participants found it easier to distinguish between negative and neutral stimuli. However, no significant interaction effects were observed between image type and group for either the positive or negative groups. For accuracy, significant differences emerged only between positive and neutral image types. While a downward trend in accuracy was observed after the task, the decrease did not reach statistical significance. Furthermore, no significant interaction effects between image type and group were detected in the accuracy data. This suggests that while fatigue influenced reaction times, its impact on accuracy was less pronounced.
| Group positive RT | Group negative RT | Group positive Acc | Group negative Acc | |||||
| F | p-value | F | p-value | F | p-value | F | p-value | |
| Group | 6.14 | 0.016* | 6.97 | 0.011* | 1.255 | 0.267 | 0.017 | 0.898 |
| Condition | 0.65 | 0.424 | 27.90 | 27.10 | 3.466 | 0.068 | ||
| Interaction | 0.15 | 0.70 | 2.525 | 0.118 | 0.017 | 0.897 | 7.231 | |
* means p
To better assess pilots’ emotional and fatigue states during simulated flight, we accounted for significant individual differences, as each participant’s classification thresholds for fatigue and emotion vary. We applied an Support Vector Machine (SVM) model to classify emotional and fatigue states using EEG, ECG, eye movement, and behavioral features. A radial basis function was used as the kernel for our classifier, and data normalization to the [0,1] range was performed using Z-scores. The dataset was split into training and testing sets with a 9:1 ratio, and the classification performance was evaluated using ten-fold cross-validation. In the fatigue classification task, labels are defined by inter-group conditions, with “before the visual search task” and “after the visual search task” serving as the two labels. In the emotion classification task, labels are defined by intra-group conditions, which are determined by the type of images presented. We assign label 1 to positive/negative images and label 0 to neutral images, to distinguish between positive/neutral or negative/neutral conditions.
We assessed the fatigue state and emotional image recognition in both the positive and negative groups during the emotional task module. The specific evaluation results are presented in the table. “Fatigue-po/ne” and “Fatigue-neutral” represent the classification of fatigue states when recognizing positive/negative and neutral images before and after the visual search task. “Emotion-pre” and “Emotion-after” denote the classification accuracy for distinguishing emotional images in each group.
Our evaluation model achieved excellent performance in classifying participants’ fatigue states, with accuracy exceeding 0.93. However, the model performed less effectively in distinguishing emotional states, achieving an accuracy of approximately 0.7. The specific results are shown in Table 8. Paired t-tests on emotion classification: p = 0.13 (positive group) and p = 0.69 (negative group) for positive/neutral images before/after fatigue task, no significant differences.
| Fatigue-po/ne | Fatigue-neutral | Emotion-pre | Emotion-after | |
|---|---|---|---|---|
| Group positive | 0.97 | 0.98 | 0.64 | 0.69 |
| Group negative | 0.94 | 0.93 | 0.67 | 0.68 |
The study investigated whether the simulated flight task successfully induced mental fatigue and examined changes in participants’ positive and negative emotional states following fatigue. It also explored the effects of fatigue on various physiological measures. To achieve these goals, firstly, we designed a simulated flight task based on visual search paradigms. Visual search tasks are used to simulate instrument scanning behaviors during flight. This scanning constitutes a core element of pilot workload during instrument flight, with studies showing that pilots allocate significant attentional resources to visual scanning under instrument flight rules [43, 44]. Visual search paradigms are also widely adopted in aviation human factors research to evaluate cockpit layout optimization [44], display complexity [45], and task load [46]. Furthermore, cross-domain validation in transportation research demonstrates the effectiveness of visual search tasks for evaluating performance and interface design, such as driver attention study under static and dynamic visual conditions [47]. Secondly, the study applied a range of analytical methods, including ERP, EEG wavelet energy analysis, eye-tracking features, ECG characteristics, and behavioral performance. Additionally, an SVM model was developed to classify participants’ emotional and fatigue states based on these measures.
Subjective scales confirmed that the visual search task effectively induced fatigue, evidenced by significant decreases in alertness and vigor and increases in anxiety and drowsiness [48]. These results align with previous findings that prolonged cognitive effort in attention-demanding tasks leads to subjective fatigue and emotional distress [49].
Behavioral performance showed significant increases in reaction time, consistent with typical indicators of mental fatigue [37, 42]. Interestingly, while accuracy showed a decreasing trend, it did not reach statistical significance, suggesting that fatigue primarily affects reaction time, while accuracy may be more resilient, possibly influenced by participants’ learning effects. Paired t-test results showed significant differences in pupil diameter, indicating that the visual search task successfully induced mental fatigue in participants [50, 51].
After the task, the heart rate significantly decreased, while TP and TINN significantly increased, reflecting reduced sympathetic activity and enhanced parasympathetic activity. The NN increased, along with increases in SDNN and RMSSD, indicating greater fluctuations in autonomic nervous system activity [44]. Although pNN50 and LF/HF also showed an increasing trend, they did not reach statistical significance. These changes in physiological indicators suggest that participants entered a state of mental fatigue, consistent with the findings from the eye-movement indicators, the subjective questionnaire responses, and EEG-related measures in this study [29, 30].
At the ERP level, N1 is an early perceptual component that reflects the initial allocation of attention to stimuli, while the N2 component is associated with conflict detection and attentional control. P2 is linked to the early evaluation of emotional information and enhanced attention, P3 reflects the late allocation of attentional resources, and LPP (Late Positive Potential) indicates sustained emotional evaluation and memory encoding [27, 39, 52].
In the positive emotion group, both positive and neutral stimuli exhibited fatigue-related changes after the visual search task, including reduced amplitudes of the N1, P3, and LPP components at the FZ and CZ electrodes. In the negative emotion group, the amplitudes of the N1 and N2 components at FZ, N1 at CZ, and N1 and P2 at PZ were all diminished. The changes in the N1 component reflect a decline in the brain’s sensitivity to positive emotional picture stimuli, reduced allocation of early attentional resources, and slower information processing speed [52]. The reduction in the P3 component suggests that after prolonged visual search tasks, participants allocated fewer attentional resources to emotional stimulus judgment, indicating potential difficulties in effectively processing emotional stimuli or regulating emotions [27, 52]. The alterations in the LPP component indicate blunted responses to emotional stimuli, impaired deep evaluation of emotional stimuli, and diminished emotional regulatory effects of positive emotional pictures [27, 39]. These findings align with previous research highlighting that fatigue impairs cognitive resources, leading to declines in attention and processing efficiency [53, 54].
Regarding condition effects, in the positive emotion group, image type significantly influenced the LPP and P3 components at FZ and CZ, as well as the P2, N2, LPP, and P3 components at PZ, suggesting that positive image stimuli had significantly stronger effects on cognitive processing and emotional stimulus intensity than neutral stimuli. In the negative emotion group, negative image stimuli exerted significantly stronger effects on cognitive processing and emotional stimulus intensity than neutral stimuli in the N2 component at FZ and CZ, and the P3 component at PZ. Notably, for the N1 and LPP components, neutral picture stimuli elicited larger amplitudes than negative picture stimuli in the negative emotion group, which differed from the positive emotion group. This phenomenon may be related to participants’ intentional inhibition of emotional responses to unpleasant pictures, which significantly reduced LPP amplitude, with this effect persisting throughout the LPP window [55].
To compare the arousal effects of positive and negative emotional stimuli on fatigue, a cross-group comparable emotional arousal index (
It has been confirmed in previous study that
The SVM-based classification model demonstrated excellent performance in distinguishing fatigue states, with accuracy exceeding 93%, consistent with the observed physiological and behavioral changes. However, its ability to classify emotional states was less effective, with an accuracy of approximately 70%. The accuracy of emotion classification was lower than that of fatigue classification. In our experiment, we only used time-domain features (such as amplitude) of EEG and frequency-domain features from wavelet decomposition, which may not be sufficient to reflect the key neural mechanisms of emotional changes. This is fewer features than those extracted in previous study on emotion classification [58]. Additionally, deep learning is currently the dominant algorithm for emotion classification, and traditional SVM has less advantage in classification accuracy. The similar classification results of positive/negative and neutral emotional stimuli before and after the fatigue task may be because the decline in emotional regulation ability due to fatigue is holistic, but it does not affect emotional responses themselves [27, 59]. In real-world scenarios such as long-duration driving, where fatigue and emotion interact, the risk of accidents is elevated, and understanding the underlying mechanisms can offer methods for algorithm optimization [60].
While this study provides valuable insights, several limitations should be acknowledged. First, the sample size of 30 participants may limit the generalizability of the findings, particularly to professional pilots who operate in highly dynamic environments. Second, the simulated flight task focused on instrument reading, which simplifies the complexities of real-world aviation scenarios (e.g., environmental stressors, multitasking demands). Third, although emotional induction tasks were effective, categorizing emotions into only three valence categories (positive, negative, and neutral) may not fully capture the multidimensional nature of emotional states. Additionally, the predefined stimuli might not reflect individual variability in emotional responses, and the short duration of emotional task blocks could have constrained the induction of sustained emotional states. Fourth, while the interaction between fatigue and emotions was explored, the specific effects of distinct emotional subtypes (e.g., anger vs. anxiety) on fatigue arousal remain unclear and warrant further investigation. Lastly, although the SVM model was effective in classifying fatigue, its accuracy in detecting emotional states under fatigue conditions was lower, highlighting the need for further research to improve classification precision in such scenarios.
In this study, a simulated flight task based on the visual search paradigm was designed. Emotional image stimuli were used to evoke participants’ positive and negative emotional states, while the visual search task was employed to induce mental fatigue. The changes following positive and negative emotional stimuli were analyzed based on EEG, ECG, eye-tracking metrics, and task performance. Additionally, the impact of these emotional states on fatigue recovery was evaluated. Finally, an SVM classifier was used to assess emotional stimuli and fatigue tasks.
The results demonstrated that the simulated flight task effectively induced mental fatigue. Following the task, participants exhibited reduced pupil diameter, prolonged reaction times, decreased HR, and increased values of NN, SDNN, and RMSSD. Neurophysiological measurements revealed that fatigue impaired attentional resource allocation and emotional processing. ERP components N1, P3, and LPP decreased, indicating weakened early attention, late cognitive evaluation, and emotional regulation capabilities. Additionally, the elevated relative energy of
ERP, event-related potential; EEG, electroencephalogram; ECG, electrocardiogram; SVM, support vector machine; HR, heart rate; HRV, heart rate variability; RT, reaction Time; Acc, accuracy; NN, mean normal-to-normal interval; SDNN, standard deviation of NN intervals; TP, total power; pNN50, percentage of NN intervals
The data generated and analyzed during the current study are not publicly available for ethical reasons but are available from the corresponding author on reasonable request.
Conceptualization, investigation, formal analysis, methodology, software, validation, writing—original draft preparation, visualization, RZ; resources, data curation, visualization, PZ and YG; formal analysis, writing—review and editing, supervision, JL, and JQ; conceptualization, project administration, funding acquisition, writing—review and editing, supervision, JB. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript. All authors have participated sufficiently in the work and agreed to be accountable for all aspects of the work.
This study was performed in accordance with the Declaration of Helsinki. This human study was approved by Biological and Medical Ethics Committee of Beihang University - approval: BM20250004. All adult participants provided written informed consent to participate in this study.
We are grateful to all the people (except the authors of this article) who took part in this study.
This research received no external funding.
The authors declare no conflict of interest.
References
Publisher’s Note: IMR Press stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.