Control over self and others’ face: exploitation and exploration
Wen Wen, Jie Mei, Hakan Aktas, Acer Yu-Chan Chang, Yosuke Suzuishi, and Shunichi Kasahara
Scientific Reports 2024
The collaboration project with Wen Wen, Hakan Aktaş, Jie Mei, Acer Chang, Yosuke Suzuishi.
Abstract
The face serves as a crucial cue for self-identification, while the sense of agency plays a significant role in determining our influence through actions in the environment. The current study investigates how self-identification through facial recognition may influence the perception of control via motion. We propose that self-identification might engender a belief in having control over one’s own face, leading to a more acute detection and greater emphasis on discrepancies between their actions and the sensory feedback in control judgments. We refer to the condition governed by the belief in having control as the exploitation mode. Conversely, when manipulating another individual’s face, the belief in personal control is absent. In such cases, individuals are likely to rely on the regularity between actions and sensory input for control judgments, exhibiting behaviors that are exploratory in nature to glean such information. This condition is termed the explorative mode. The study utilized a face-motion mixing paradigm, employing a deep generative model to enable participants to interact with either their own or another person’s face through facial and head movements. During the experiment, participants observed either their own face or someone else’s face (self-face vs. other-face) on the screen. The motion of the face was driven either purely by their own facial and head motion or by an average of the participant’s and the experimenter’s motion (full control vs. partial control). The results showed that participants reported a higher sense of agency over the other-face than the self-face, while their self-identification rating was significantly higher for the self-face. More importantly, controlling someone else’s face resulted in more movement diversity than controlling one’s own face. These findings support our exploration–exploitation theory: When participants had a strong belief in control triggered by the self-face, they became highly sensitive to any sensorimotor prediction errors, leading to a lower sense of agency. In contrast, when the belief of control was absent, the exploration mode triggered more explorative behaviors, allowing participants to efficiently gather information to establish a sense of agency.
Does seeing your own face strengthen control?
When people look in a mirror, they have no doubt that they can control the reflection: recognizing oneself comes with a belief in control. Does that belief make the sense of agency stronger? A study with people’s own voices suggested that it does. The sense of agency, however, also depends on how sensitive people are to mismatches between their actions and what they perceive, and how a belief in control affects that sensitivity had not been examined. We asked how recognizing one’s own face shapes the sense of agency over it, and the movements people make while judging it.
Moving one’s own face or someone else’s
Using a deep generative model (Latent Image Animator), the face on the screen followed the participant’s head and facial movements in real time, rendered either with the participant’s own face or with the experimenter’s face. In the full control condition, the face followed only the participant’s motion; in the partial control condition, it followed an average of the participant’s and the experimenter’s motion. The same algorithm generated every video, so the actual level of control did not depend on whose face was shown. Twenty participants moved the face freely for 20 seconds in each trial and then rated how much the face resembled their own and how much control they felt over it. Even with full control, the video lagged about 300 ms behind their movements, so small discrepancies remained in every condition.
Exploitation and exploration
Recognizing their own face put participants in an exploitation mode, and someone else’s face in an exploration mode: they reported less control over their own face, and they moved someone else’s face more widely and in more varied ways. Participants identified with their own face far more than with the other face, yet their agency ratings were higher for the other face, with both full and partial control (main effect of face, F(1, 19) = 11.42, p = 0.003). This was not because they controlled the other face better: the measured motion errors were, if anything, slightly smaller for their own face. Their movements differed as well. With someone else’s face, the head, eyes and lips all moved over larger distances, and the diversity of their motion was higher (F(1, 19) = 5.56, p = 0.029).
Why one’s own face lowers the sense of agency
The paper explains these results with its exploration–exploitation theory. Recognizing one’s own face brings the belief that one is in control, as with a mirror. Under that belief, people exploit their control, and their perception is tuned to prediction errors, so small mismatches between action and feedback, here the lag and spatial errors of the generated video, weigh heavily and the sense of agency drops sharply. Without the belief, as with someone else’s face, people look for regularities between their actions and what they see, and explore with larger and more varied movements to gather evidence of control; their sense of agency rises more gradually and tolerates small errors. The belief in control that comes with self-identification therefore did not enhance the sense of agency, as the earlier voice study had suggested, but made it more fragile.
The paper also notes an alternative it cannot rule out: people may simply be better at detecting mismatches on their own, familiar face, and the study cannot separate a stronger weighting of errors from better detection. Only with perfect control, which this setup could not provide, would the theory expect more confidence in control over one’s own face.
Avatars that look like us
The findings suggest that a control interface that resembles the user too closely, such as a humanoid robot or a virtual avatar, may reduce the sense of agency, which the paper likens to the uncanny valley. As people come to control more avatars and robots through human–machine interfaces, the theory may help predict when they will feel in control and when they will explore. The paper presents the study as an initial test of the theory, based on subjective ratings.
Reference
Wen Wen, Jie Mei, Hakan Aktas, Acer Yu-Chan Chang, Yosuke Suzuishi, and Shunichi Kasahara. 2024. Control over self and others’ face: exploitation and exploration. Scientific Reports 14, 1, Article 15473 (July 2024). https://doi.org/10.1038/s41598-024-66316-2
BibTeX
@article{10.1038/s41598-024-66316-2,
author = {Wen, Wen and Mei, Jie and Aktas, Hakan and Chang, Acer Yu-Chan and Suzuishi, Yosuke and Kasahara, Shunichi},
title = {Control over self and others’ face: exploitation and exploration},
journal = {Scientific Reports},
volume = {14},
number = {1},
year = {2024},
articleno = {15473},
publisher = {Springer Science and Business Media LLC},
doi = {10.1038/s41598-024-66316-2}
}Figures from the paper (Wen et al., Scientific Reports 2024), licensed under CC BY 4.0.
- News: The new paper on the collaboration project from Nature Scientific Reports
- Related projects: Face boundary in action