Haptic exploration for extracting information about objects is generally considered a silent process. When vision is absent, the texture of an object is thus perceived through multisensory processing of tactile and proprioceptive information (i.e., haptic information). Interestingly, when haptic exploration becomes audible, the perception of texture can change. One of the most compelling demonstrations of tactile-auditory integration altering texture perception is the so-called parchment-skin illusion [1, 2]. Specifically, people experiencing this illusion perceive the palms of their hands as becoming either dry or moist, depending on the spectral content of the hand-friction sound played while rubbing hands together. Thus, adding movement-related sounds, a technique called movement sonification, can significantly alter the perceived texture, even though the tactile and proprioceptive inputs evoked during surface exploration remain unchanged. The primary objective of this technique, which is conceptually like the Foley effect in cinema, is to convey movement-related information and enhance both the perception and control of movement [3]. Movement sonification may compensate for the absence or reduction of sensory input, supporting hypotheses of auditory-proprioception substitution [4] and auditory-visual substitution [5]. Potential clinical applications for motor rehabilitation are therefore numerous: for example, individuals who are blind, use a limb prosthesis, have suffered a stroke, or present peripheral nerve injuries may benefit. Applied to pen movements, movement sonification has already been successfully used to improve handwriting performance [6, 7]. In texture discrimination tasks, haptic exploration has been shown to change in the presence of textured sounds compared to non-textured sounds, notably by increasing movement velocity [8].

A fundamental property of sound is its capacity to evoke a wide range of emotions [9, 10]. Despite this, little is known about how auditory-evoked emotions influence the appraisal of textured surfaces scanned by a finger. This gap persists although there is well-established evidence that our appreciation of objects is strongly influenced by the tactile sensations and emotions elicited when touching them. The present study aims to determine whether movement sonification during tactile exploration of a surface alters both the perceptual appraisal of the texture and, its emotional valence (ranging from “positive, pleasure” to “negative, displeasure”).

Changes in the perceptual and emotional appraisal of a texture can result from bottom-up and top-down mechanisms. From a bottom-up perspective, the Bayesian approach proposes that the central nervous system integrates information from different sensory modalities based on the estimated probability that it originates from a common or independent source [11]. The mechanisms underlying cross-modal interactions in the human brain remain largely unknown. This is particularly the case for audio-tactile interactions, despite their frequent occurrence in everyday life, for example, when hearing footsteps on a gravel path. This example complies with the “temporal principle” of multisensory integration stating that an interaction is most likely to be maximally effective when stimuli overlap in time [12]. Sound and touch are likely to respond to this principle as they are deeply entangled. Mechanical forces generated by physical contact not only activate somatosensory mechanoreceptors but also generate acoustic waves detectable by the auditory system [13]. To date, neurons responsive to both audio and tactile stimuli have been found in primates in the ventral premotor cortex, the ventral intraparietal region and the superior temporal sulcus [14, 15]. Further evidence indicates that audio-tactile interactions occur already at early stages of sensory processing. Notably, the somatosensory (S1) and auditory (A1) cortical regions, presumptive unimodal sensory areas, exhibit enhanced responses to bimodal stimuli relative to unimodal inputs [16]. This has been evidenced in animal studies by Lakatos et al. [17] showing that somatosensory inputs reset the phase of ongoing neuronal oscillations, so that accompanying auditory inputs arrive during an ideal, high-excitability phase, and produce amplified neuronal responses. In contrast, responses to auditory inputs arriving during the opposing low-excitability phase tend to be suppressed. Zhang et al. [18] extended these findings from primary cortical areas to secondary somatosensory cortice (S2). They showed in the mouse neocortex that, pairing sound with whisker stimulation modulates tactile responses in both S1 and S2. Sound responses of many sound-selective neurons in S2 neurons were spatially colocalized with S2 touch-selective neurons.

Complementing these bottom-up mechanisms, other approaches propose that the perception of a texture under our skin is not driven solely by somatosensory stimulation [19]. Rather, somatosensory processing would be under the influence of top-down control, which actively modulates, and routes sensory inputs based on expectation, attention, and the perceptual task [20, 21]. For example, pleasure induced by listening to pleasant music has been shown to require higher levels of attention compared to neutral music, thereby reducing the attentional resources available for other tasks [22]. This relationship between attention and the experience of pleasurable sounds can influence how sensory information is processed and integrated, as attentional states have been shown to modulate somatosensory processing [23].

Through top-down and bottom-up mechanisms, movement sonification may then modulate the hedonic valence of textures during haptic exploration, particularly when the associated sounds convey emotional attributes. In the present study, we hypothesized that texture exploration producing pleasant sounds through movement sonification would be rated as more pleasant, while textures associated with unpleasant sounds would be perceived as less pleasant. These perceptual effects of movement sonification may be linked to changes in beta power within cortical areas involved in processing somatosensory inputs (e.g., S1, Posterior Parietal Cortex (PPC)) as well as in regions engaged in evaluating emotional valence, such as the cingulate cortex. This hypothesis is consistent with previous work showing that beta-band (15–25 Hz) oscillations can be used to probe the relationship between cortical functioning and emotional dimension [24]. In line with this, studies have reported stronger beta response to unpleasant materials (e.g., pictures [25] or video [26, 27]). For example, using video, Mishra et al. [26] found higher fronto-temporo-parietal connectivity pattern that captured emotion specifically in the beta band, compared to other frequency bands.

The aim of this study was to investigate whether movement sonification alters the perceptual evaluation and hedonic valence of textured surfaces, as assessed through behavioural, psychophysical, and neurophysiological measures. The results show that texture appraisal is impervious to sonification conditions, whereas their emotional valence is not. This effect was observed with the P2 rough surface, which was classified as disagreeable and perceived as even more unpleasant when paired with disagreeable movement sonification. However, the brain’s response to the audio-tactile integration was altered by both the agreeable and the disagreeable movement sonifications. In both conditions where stroking movement evoked auditory inputs, we observed a marked decrease in beta-band oscillatory power over the sensorimotor cortex and the superior parietal cortex as compared to silent touch. Such decreases in beta-band power are commonly interpreted as reflecting increased cortical excitability and enhanced sensory processing [36, 37, 38, 39]. It is also in line with the functional magnetic resonance imaging (fMRI) study by Landelle et al.’s [16], which demonstrated the sensitivity of the primary somatosensory cortex to tactile input from the hand. The reason for the increased sensorimotor activity requires further investigation, and may reflect a combination of motor, emotional, and attentional processes. Nevertheless, our findings suggest that multisensory interactions are not confined to higher-order brain regions but also occur in early somatosensory areas, including primary somatosensory cortices and superior parietal regions.

This suggestion is supported by neuroanatomical studies showing direct thalamocortical projections to the superior parietal areas in macaques [40, 41, 42, 43]. The early multisensory interaction is thought to be mediated by means of phase resetting. The presentation of a stimulus to one sensory modality (e.g., auditory) resets the phase of ongoing oscillations in another modality (e.g., tactile) such that processing in the latter modality is modulated [17, 44].

While these data speak to the importance of primary somatosensory regions for audio-tactile information integration, evaluating the hedonic valence of movement sonification likely engages higher-order brain areas. One candidate region is the PCC, which exhibited a significant increase in beta band ERD (15–25 Hz) during agreeable sonification compared to silent touch. However, this change in beta band power was not accompanied by a corresponding change in the emotional rating of the scanned surface. It is possible that altering the hedonic valence of the touched surface would require a sound that is not merely free of dissonance but is explicitly pleasant. The fact that PCC beta-band power remained unchanged during disagreeable sonification suggests that this medial cortical region may be specifically involved in processing pleasantness. In this light, our findings are consistent with studies showing that the PCC is engaged in evaluating the emotional valence [45] and context of sensory information via pathways from somatosensory regions (superior and inferior parietal areas [46]), auditory regions [47] and adjacent cingulate areas such as the midcingulate cortex [48, 49, 50]. Due to these extensive connections, the PCC is considered to exhibit the highest degree of cross-network interaction [51].

When false notes were present while stroking the P2 rough surface, a bilateral decrease in activity was observed in the early visual cortex compared to silent touch. This was evidenced by an increase in event related synchronization ERS (red/warm color, Fig. 3A, right panel) in occipital regions (BA17/18). One could argue that this effect reflects task-related disengagement, given that participants’ eyes were closed. However, because the eyes were closed in all sonification conditions, the increased ERS observed in visual areas when contrasting beta power topography between the disagreeable and silent conditions cannot be attributed to the absence of visual feedback. The disengagement of visual cortical areas during disagreeable movement sonification may have played a crucial role in enhancing auditory-driven emotional responses. Surface texture is often co-deployed with a visual pattern, allowing the formation of a combined haptic-visual representation of the surface. The reduction in visual cortex activity may have redirected participants’ attention away from the visual attributes of the surface, thereby giving more weight to the disagreeable sonification and amplifying its emotional impact.

The potential mechanism explaining why participants rated the rough surface as even more unpleasant when their haptic exploration was paired with dissonant sonification is twofold. First, the visual cortical area, known to receive direct retinal inputs from the thalamic lateral geniculate nucleus, can be modulated by auditory signals. Signatures of auditory inputs in primary visual cortex were previously observed by Vetter et al.’s [52] fMRI study in absence of visual inputs [53]. Their results showed that the visual cortex contributes, via its corticocortical connections, to the encoding of complex natural sounds as abstract, higher-level internal representations [54], further highlighting the existence of multiple cortical representations of sounds.

Second, our results may be interpreted within the framework of the Bayesian causal inference model that postulates that multisensory integration is a consequence of an optimal probabilistic inference of each sensory modality [55]. Accordingly, both sensory sources, auditory and tactile, are integrated in the judgment task but with a different weight allocated to each sensory input, which depends on the level of relevance and accuracy of the processed information. For instance, for the surface P2, the addition of disagreeable sounds shifts the emotional response towards the unpleasant, as the emotional categorization of the tactile modality is less stabilized as shown by the emotional rating distribution. The lack of effect of sound conditions on neural excitability for the P1 slippery and agreeable surface suggests that the tactile modality has a greater weight than the auditory modality in this texture perception task, both at the perceptual and emotional levels. In other words, adding an unpleasant sound is not sufficient to override the agreeable tactile experience (P1). This observation may also point to the presence of a valence threshold effect; whereby only unpleasant textures are subject to modulation. In line with this interpretation, Ackerley et al. [56] reported that listening to classical music evoking sad, neutral, or happy emotions alters the sense of limb position (i.e., proprioception, not tactile), by increasing muscular fiber sensitivity to muscle shortening/lengthening. This effect was found for sad music, but not for happy music, relative to neutral, further suggesting a valence threshold effect.

This study shows that the hedonic valence of auditory signals can modulate emotional aspects of touch without altering texture perception and, outlines possible underlying brain mechanisms. However, it presents some limitations. The most notable concerns the relatively small sample size and the limited set of textured surfaces tested which can both restrict the generalizability of the findings. Future studies should also consider individual differences in sensory (audio, tactile) discrimination abilities and in the emotion elicited by touch and sound. Indeed, we know that sensitivity to sounds and music [57], and to haptic information [58] varies across individuals. Exploring both sensory domains in tandem would provide a more comprehensive understanding of the complex interplay between tactile and auditory processing.

Our results offer new insights into the neural mechanisms by which hedonic valence of auditory signals modulates haptic emotional processing, without disrupting the perceptual analysis of texture properties. More precisely, texture appraisal seems unaffected by sonification conditions, whereas their emotional valence is clearly influenced. This effect was evident for the disagreeable texture, which was perceived as even more unpleasant when paired with disagreeable sonification. At the neural level, audio-tactile integration was altered by both the agreeable and the disagreeable movement sonifications. These findings confirm that the emotional valence of the texture perception arises not only from tactile input, but also from multisensory processing. Finally, augmented audio reality offers promising clinical applications for modulating the emotional and sensory experience when interacting with physical objects in individuals with sensory or emotional regulation deficits.

The paper is listed as, “Movement sonification during haptic exploration shifts emotional outcome without altering texture perception” as a preprint on BioRxiv at: https://www.biorxiv.org/content/10.1101/2025.05.28.656569v1.abstract.

The data generated and analyzed during the present study are not publicly available for ethical reasons but are available from the corresponding author on reasonable request.

Conception, investigation, formal analyses, methodology: LM, PHC, JF, AW, FM, EC, JD designed the research, analyzed and interpreted the results. CS, BW, JB analyzed the data and interpreted the results. LM, PHC, JF, AW, FM, EC, JD, JB wrote the manuscript. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript. All authors have participated sufficiently in the work and agreed to be accountable for all aspect of the work.

All participants gave their written informed consent to take part in this study, which conformed to the standards set in the Declaration of Helsinki. This study was approved by the Ethics Committee for research in Sciences et Techniques des Activités physiques et Sportives (CERSTAPS: IRB00012476-2021-09-12-140).

This work was supported by the project COMTACT (ANR 2020-CE28-0010-03), funded by the french “Agence Nationale de la Recherche” (ANR).

The authors declare no conflict of interest.