There is a current increase of interest in the relationship between facial expressions and autobiographical memory (i.e., memory of personal experiences or events) that are related to the self [1,2]. Much of this interest originates in the discovery that autobiographical retrieval can trigger emotional facial expressions. This issue previously been investigated in a study that demonstrated emotional facial expressions during retrieval of emotional autobiographical memories [3]. In that study, participants were invited to recall memories related to emotional cues and retrieval was analyzed with facial analysis software (Facereader). This software applies an algorithm to locate a face in images or videos; it also synthesizes the face by calculating the location of 500 key points and includes the facial texture of the area delineated by these 500 points [4,5,6]. The software successively classifies facial expressions using an artificial neural network that classifies basic emotions described by Ekman [7,8] (i.e., neutral, happy, sad, surprised, angry, scared, and disgusted). The software classifies emotional expressions by specifying their facial action units, which typically represent a distinct movement of the face that is isolated from other parts of the face [9,10].

The evaluation of facial expressions during autobiographical retrieval was also investigated in a study that assessed past and future thinking [11]. In that study, subjects remembered and imagined future personal events. Subjects demonstrated more emotional and fewer neutral facial expressions during imagining the future than during remembering the past. By describing facial expressions during future thinking, El Haj and Antoine [11] provide physiological evidence on the emotional characteristics of mental time travel into the future [12, 13, 14, 15, 16]. The same thing can be said for the evaluation of facial expressions during autobiographical retrieval in general; this evaluation provides valuable physiological information about the emotional experience of the past as this experience has been mainly evaluated with subjective methods (e.g., scales on which participants indicate their emotional state) [17, 18, 19, 20, 21, 22, 23, 24].

To provide further insight about emotional facial expressions that may be triggered by autobiographical retrieval, the present study investigates facial expressions during the retrieval of self-defining memories (SDMs). SDMs are important as these memories are highly associated with the self and identity [25, 26]. SDMs refer to those vivid, emotionally intense, and repetitively recalled memories of enduring concerns or unresolved conflicts [27]. According to Conway and Singer [28], these memories help maintain self-consistency and self-coherence, particularly during times of difficult transitions or upheaval. Supporting this assumption, studies have demonstrated the involvement of SDMs in the anticipation of important future events [29], the pursuit of long-term goals [30], development of personality [31], activation of specific self-representations [32], and emotional regulation [33, 31, 34]. At a social level, people frequently describe themselves and their life stories by sharing SDMs with others as this sharing facilitates intimacy and the transmission of personal experience [35, 36]. Beyond their social-sharing function, SDMs contribute to meaning making and life story integration [25, 37, 36]. In other words, these memories help people establish the meaning of past events and integrate this meaning within the context of their life story [38, 39].

The importance of SDMs to the self and identity has been illustrated with an integrated model of narrative identity by Singer and Blagov [27], according to whom such memories contribute to the creation of narrative scripts that constitute a stock of "chapters" across one's life story. These assumptions of Singer and Blagov [27] have been supported by studies demonstrating the relationship between compromise of the ability to retrieve SDMs and diminished sense of self and identity in psychiatric [40, 29, 41, 42, 43, 44] and neurological [44, 45] populations. Together, SDMs are memories that significantly contribute to the sense of self and identity. By investigation of facial expressions during retrieval of SDMs, this study ought to provide physiological insight into the emotional salience of these kind of memories.

To assess facial expressions that may be triggered by retrieval of SDMs, subjects were invited to retrieve memories that significantly contributed to their life story and identity. Retrieval was video recorded and later analyzed with facial analysis software. Besides this objective evaluation, subjects were invited to evaluate the emotional valence of their memories on a self-rating scale. It was predicted that participants would show more emotional than neutral facial expressions during SDM retrieval.

To investigate the hypothesis (i.e., whether SDMs would trigger more emotional than neutral facial expressions), the percentage of total facial emotional expressions (i.e., percentage of the six emotional expressions: happy, sad, angry, surprised, scared, disgusted) were compared with the percentage of neutral expressions during retrieval of SDMs. The percentage of positive (happy and surprised) vs. negative facial expressions (sad, angry, scared, disgusted) were also compared, as this categorization has been proposed by Ekman [8]. The subjective emotional experience of SDMs were also analysed, as rated by the subjects on the Self-Assessment Manikin scale. Non-parametric tests were employed because the subjective evaluation of emotion was scale, and also because of the non-normal distribution of percentage of some facial expressions, the latter determined by Kolmogorov-Smirnov tests. Comparisons were performed using the Wilcoxon signed rank test. The relationship between the subjective emotional evaluation of SDMs and emotional facial expressions were evaluated by regression analysis. For all tests, significance was set at p $\leq$ 0.05 . Effect sizes were reported for signifcant differences; d = 0.2 can be considered a small effect size, d = 0.5 represents a medium effect size and d = 0.8 refers to a large effect size [48]. Note that Cohen's d was calculated for non-parametric tests following recommendations by Rosenthal and DiMatteo [49], and Ellis [50].

3.1. SDMs trigger more emotional than neutral expressions

The percentages of emotional and neutral facial expressions during retrieval of SDMs are described in Fig. 1. Analysis revealed that subjects showed more emotional (i.e., happy + sad + angry + surprised + scared + disgusted = 52.60 , $SD $ = 17.27 ) than neutral (M = 39.05,$SD $=20.63) facial expressions during retrieval of SDMs ($Z$ = 2.13, p < 0.05, Cohen's d = 1.08). Also that participants were more positive (i.e., happy + surprised facial expressions = 33.45 , $SD $ =12.67) than negative (sad + angry + scared + disgusted = 19.15 , $SD $ = 18.81 ) for facial expression during retrieval of SDMs ($ Z$ = -3.52, p < 0.001 , Cohen's $ d = 2.55 $). Subjects also showed more happy than sad ($ Z $ = -3.49, p < 0.001, Cohen's d = 2.49), angry ($ Z$= -3.55 , p < 0.001 , Cohen's d= 2.61), scared ($ Z$ =-3.74 , p< 0.001, Cohen's d = 3.05 ), or disgusted ($ Z$ = -3.61 , p< 0.001, Cohen's d = 2.73 ) facial expressions during retrieval of SDMs. The were also more surprised than sad ($ Z $ = -3.38, p< 0.001 , Cohen's d = 2.30), angry ($ Z $= -3.57, $ p< 0.001 $, Cohen's d = 2.62), scared ($ Z $= -3.72, p < 0.001, Cohen's d= 1.53), or disgusted ($ Z $= -3.58 , p< 0.001, Cohen's d= 2.67) by facial expression during retrieval of SDMs. All other comparisons were not significant (p > 0.10).

Fig. 1.

Facial expressions during retrieval of self-defining memories. Note. The state "unknown" refers to situations in which the face could not be modelled.

3.3. Facial expressions predict subjective experience

Relationship between the subjective emotional evaluation of SDMs and emotional facial expressions was investigated by regression analysis. The dependent variable was the mean rating on the Self-Assessment Manikin scale whereas the predictor variables were the positive emotional score of the Facereader (i.e., the sum of happy and surprised facial expressions) for each SDM, therefore five predictor variables (corresponding to five SDMs) were entered simultaneously for regression analysis. Analysis showed that the positive emotional score of the Facereader software significantly contributed to the score on the Self-Assessment Manikin scale, accounting for 36.60% ($ p< 0.05 $) of its variance (see Fig. 2).

Fig. 2.

Regression plot, illustrating the relationship between the subjective emotional experience of self-defining memories (SDMs), as rated by the participants on the Self-Assessment Manikin scale, and positive emotional facial expressions during retrieval of each SDM.

Facial expression during retrieval of SDMs were investigated. Facial analyses software made it possible to show that the facial expressions of subjects were more emotional than neutral, as well as more positive than negative during SDMs retrieval. It was also found that the subjective experience of SDMs was positive, as rated by subjects on the Self-Assessment Manikin scale. These findings demonstrate the consistency between facial expression and emotional subjective experience of SDMs.

Investigators have long recognized the intimate connection between SDMs and the construction of identity [25, 26]. Research has emphasized the centrality of such memories to the functioning of identity, viewing them as a window into the development of identity and evolution of individual identity [25, 39, 37, 36]. Research has also emphasized the emotional intensity, vividness, and linkages to other memories [33, 31, 34], as well as the centrality of SDMs for self-images, self-understanding, and unresolved conflicts or enduring concerns [28]. SDMs are believed to reflect the dominant concerns in an individual's present life situation and, in moments of uncertainty, they serve as a landmark to remind the individual of his/her identity [25]. Interestingly, emotions associated with self-defining memories were found to be related with changes in personality, well-being, and professional performance [31]. Together, these findings highlight the centrality of SDMs to identity and personality, as well as the centrality of emotion to SDMs. The emotional characteristic of SDMs was observed in this study as data demonstrated emotional facial expressions during retrieval of these memories. The emotional facial expressions, as observed during retrieval of SDMs, can be interpreted in light of a theoretical assumption according to which SDMs are believed to have the power to evoke physiological states similar to those experienced in the encoded events [31]. In other words, emotional facial expressions, as observed in this study during retrieval of SDMs, can be considered as reflecting the affective and motivational content of the encoded events. Alternatively, these expressions can be considered as reflecting emotional regulation during retrieval of SDMs.

Besides demonstrating more emotional than neutral facial expressions during the retrieval of SDMs, findings demonstrated more positive than negative expressions during this retrieval, as well as more positive than neutral subjective evaluation of these memories. These findings mirror a study by Wood and Conway [34] who found positive evaluation of SDMs. More specifically, Wood and Conway [34] observed that, for negative SDMs, participants reported less negative and more positive emotion compared to how they recalled feeling at the time, and the same thing was observed for positive SDMs. Therefore, SDMs seem to trigger a positive emotional state, at least in subjects without psychiatric or neurological disorders. From this perspective, people would presumably describe less negative emotion during retrieval of SDMs because meaning making, a core characteristic of SDMs, involves an assumption of evolution of memory appraisal (i.e., of improved regulation of memory over time). Additionally, learning lessons or finding benefits from negative memories would presumably lead people to experience more positive emotions about the memory. According to this assumption, one study has demonstrated that life narratives trigger a transformation from negative to positive affect rather than vice versa [51]. This phenomenon of "making things better" mirrors the more general notion of redemption sequences in the life story and research about "happy endings" to negative events [52], "fading affect bias", i.e., that fading is greater for unpleasant emotions than for pleasant emotions associated with autobiographical events [53, 54] and the "wisdom of experience", i.e., wisdom to transform negative events to positive life situations, a phenomenon especially observed in adulthood [55, 56]. Another characteristic of SDMs that may be responsible for their positive valence is social sharing. As mentioned in the introduction, people frequently describe themselves and their life stories by sharing SDMs [35, 36]. When memories, especially positive ones, are shared with others, people tend to experience a positive experience that is above the original experience [57]. Together, the positive characteristics of SDMs, as observed in this study, can be attributed to meaning making (i.e., reflecting on the positive outcomes of a negative event) and/or to social sharing.

It is suggested that the positive emotional facial expressions, as observed in subjects during retrieval of SDMs, can be attributed to the retrieval of the original emotional experience and/or to the emotional regulation of the retrieved memories. The prevalence of positive emotional expressions, as reported for this study, can also be compared with a study assessing these expressions during future thinking [58]. That study found higher emotional, as well as higher positive facial expressions, during future than during past thinking; these findings mirror the picturesque accounts of future thinking. According to this perspective, people tend to imagine the future by setting new goals with the expectation that they will succeed [59, 60].

To further understand the variations of facial expressions during retrieval of SDMs, and memory retrieval in general, it would be of interest to assess these expressions according to emotional regulation during retrieval. As explained here, emotional reactions during retrieval may depend on how people reflect on the outcomes of their memories or even on how they define these outcomes according to their current goals are self-representations (e.g., a memory of a failure is more likely to be considered as a less positive if retrieved during a current achievement than during a current failure). Therefore, it would be of interest to investigate facial expressions during memory retrieval according to emotional regulation profiles. For instance, one may expect fewer emotional facial expressions in people experiencing emotional avoidance. Also, individuals high in agency (i.e., assertive individuals with propensity for mastery and power) may demonstrate highly emotional expressions during memory retrieval. An individual's underlying emotional motive may influence facial expression during memory retrieval. This assumption can be tested for specific personality characteristics. For instance, because intrapersonal self-regulatory processes of a narcissist tend to maintain artificially high self-esteem, their memories are likely to be saturated with specific facial expressions (e.g., contempt) than non-narcissist individuals. Beyond the personality structure, it would be of interest to assess facial expressions during retrieval of autobiographical memories in psychiatric populations. For instance, patients with depressions may demonstrate more negative than positive expressions during retrieval, which may correspond to the negative emotion valence of autobiographical retrieval in depression [61]. In addition, because SDMs tend to be negative in alcohol-dependence [40, 41, 42], it would be of interest whether retrieval of these memories would trigger negative emotional facial expressions.

As mentioned in the Introduction, the Facereader software relies on an artificial neural network to map 500 facial points of reference (on the input layer) onto a pre-set list of basic emotions described by Ekman [7,8]. However, although the model of Ekman provides a comprehensive description of basic emotional categories, there is a wide variety of emotional states that fall outside of this description. This assumption can be supported by many theories suggesting that emotional experiences extend beyond Ekman's list of basic emotions [62, 63]. Therefore, although classification by Facereader provides a description of basic-emotional categories, facial expressions during memory retrieval can be better understood if assessed with software dealing with complex emotions. Regardless of this issue, results demonstrated high levels of "surprised" states during memory retrieval. These high levels can be attributed to the fact that, prior to the retrieval instructions,subjects were informed that the study was designed to investigate facial expressions during memory retrieval and no mention was made of the self-defining nature of the memories probed for. Therefore, participants might be surprised when asked to retrieve personal memories, and more specifically, Self-related highly personal ones. Another possibility is that participants were surprised by their ability to share personal memories or even by the content of their memories.

One potential shortcoming of this study is the lack of a control condition allowing the comparison of facial expressions during SDMs and an appropriate control situation (e.g., a condition in which participants retrieve a typical autobiographical memory). Regardless of this limitation, our experimental design allowed comparisons of emotional and neutral facial expressions during retrieval of SDMs. Our experimental design also allowed comparison of objective and subjective emotional states during retrieval of SDMs.

By comparing facial expressions during retrieval of SDMs, this study demonstrates how subjective emotions experienced during recollection of highly salient and pertinent memories can be mirrored by emotional facial activity. By doing so, the study serves as a bridge between psychological research on subjective experiences of the past and physiological research on the physical and nervous activity that may be convolved with cognitive processing. it is hoped this study will pave the way to research demonstrating how description of facial activity during memory retrieval can provide insights into the cognitive mechanisms underlying memory construction.

The authors would like to thank Alicia Prudence and Pauline Boriau for assistance in recruitment of participants.

All authors declare no conflicts of interest.