Investigating the impact of motion visual synchrony on self face recognition using real time morphing
Shunichi Kasahara, Nanako Kumasaki, and Kye Shimizu
Scientific Reports 2024
Abstract
Face recognition is a crucial aspect of self-image and social interactions. Previous studies have focused on static images to explore the boundary of self-face recognition. Our research, however, investigates the dynamics of face recognition in contexts involving motor-visual synchrony. We first validated our morphing face metrics for self-face recognition. We then conducted an experiment using state-of-the-art video processing techniques for real-time face identity morphing during facial movement. We examined self-face recognition boundaries under three conditions: synchronous, asynchronous, and static facial movements. Our findings revealed that participants recognized a narrower self-face boundary with moving facial images compared to static ones, with no significant differences between synchronous and asynchronous movements. The direction of morphing consistently biased the recognized self-face boundary. These results suggest that while motor information of the face is vital for self-face recognition, it does not rely on movement synchronization, and the sense of agency over facial movements does not affect facial identity judgment. Our methodology offers a new approach to exploring the ‘self-face boundary in action’, allowing for an independent examination of motion and identity.
How much of a moving face is still mine?
We recognize our own face not only in photographs but also in a mirror or on a video call, where it moves with us. Studies of the boundary between one’s own face and another’s have mostly used static images morphed between two faces. Studies of the enfacement illusion have shown that this boundary can shift: after synchronous visuo-tactile or visuo-motor stimulation, people start to identify with another person’s face. The sense of agency, the feeling of controlling a movement, is known to go together with the sense that a body part is one’s own, but whether it also affects whether a face looks like one’s own had not been tested. Part of the reason is technical: it was hard to change the identity of a moving face while keeping its movement natural. We asked whether a moving face has to move in sync with us to be recognized as ours.
Morphing a face in real time
We first built a morphing scale. For each participant, an “other” face was generated at a fixed distance from their own face in the latent space of a StyleGAN2 network, so that morphing from self to other spans a comparable distance for everyone. In a first experiment, participants rated static morphs toward six target faces, and the boundary between self and other always fell inside the morphing range (on average at 44%), which validated the scale.
In the second experiment, the morphing face moved with the participant, like a reflection in a mirror. A camera captured the participant’s face, and an animation network applied its movements to the morphed face at about 30 frames per second, with a delay of about 250 ms. The face morphed by 1% per second, either from self to other or from other to self, while the participant turned their head slightly every two seconds and pressed a key as soon as the face looked “more like other than self” (or the reverse). Three conditions differed only in how the face moved: in Sync, as described; in Async, with a delay that varied randomly between 0.32 and 2.5 seconds; and in Static, not at all. Data from 31 participants were analyzed.
Still my face, out of sync
Even when the face did not move in sync with them, participants recognized it as their own just as much: the boundary of their own face was the same in Sync and Async, although they felt far less in control of the out-of-sync face. The difference in the self-face boundary between Sync and Async was not significant (p = 0.69), while synchronization had a strong effect on the reported sense of agency, which differed between all three conditions.
Movement itself did matter. With a moving face, in Sync or Async, participants gave the face up as their own earlier when it morphed toward the other, and accepted it later when it morphed toward themselves, than with a static image. In other words, the self-face boundary was narrower for a moving face than for a static one.
Why synchrony did not decide identity
The paper likens the task to a digital mirror test. Recognizing oneself in a mirror is thought to combine a contingency cue, the match in timing between one’s action and what one sees, and a figurative cue, the match between the seen face and the self-face stored in memory. The results suggest that the contingency cue, which drives the sense of agency, does not decide facial identity. Facial identity and the sense of facial ownership may therefore work through processes separate from the sense of body ownership, which earlier studies found to be strengthened by synchrony. The influence may even run the other way: the effect of synchronization on the agency scores differed depending on whether the morph started from the participant’s face or the other’s, suggesting that perceived identity can affect the sense of agency. The narrower boundary for a moving face may come from integrating the two cues, although the design could not separate the movement itself from the different head postures it brings.
A face that moves without me
In the Async condition the face still showed the participant’s own movements, only with random delays, and they no longer felt they were controlling it, yet they recognized it as their own as much as in Sync. Recognizing one’s moving face, in this sense, did not depend on moving it oneself. As AI-generated faces and real-time face synthesis spread, a face can be animated by someone or something other than its owner, and understanding how facial identity forms becomes a question of identity protection as well as perception. The method, which changes a face’s identity while keeping its movement, makes it possible to study this “self-face boundary in action” and to revisit findings, such as the enfacement illusion, that were obtained with static images.
Reference
Shunichi Kasahara, Nanako Kumasaki, and Kye Shimizu. 2024. Investigating the impact of motion visual synchrony on self face recognition using real time morphing. Scientific Reports 14, 1, Article 13090 (June 2024). https://doi.org/10.1038/s41598-024-63233-2
BibTeX
@article{10.1038/s41598-024-63233-2,
author = {Kasahara, Shunichi and Kumasaki, Nanako and Shimizu, Kye},
title = {Investigating the impact of motion visual synchrony on self face recognition using real time morphing},
journal = {Scientific Reports},
volume = {14},
number = {1},
year = {2024},
articleno = {13090},
publisher = {Springer Science and Business Media LLC},
doi = {10.1038/s41598-024-63233-2}
}Figures from the paper (Kasahara et al., Scientific Reports 2024), licensed under CC BY 4.0.
- News: The new paper is out from Nature Scientific Reports
- Related projects: Control over self and others’ face, Morphing Identity