Morphing Identity

Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara
ACM CHI 2023

Abstract

We explored continuous changes in self-other identity by designing an interpersonal facial morphing experience where the facial images of two users are blended and then swapped over time. Both users’ facial images are displayed side by side, with each user controlling their own morphing facial images, allowing us to create and investigate a multifaceted interpersonal experience. To explore this with diverse social relationships, we conducted qualitative and quantitative investigations through public exhibitions. We found that there is a window of self-identification as well as a variety of interpersonal experiences in the facial morphing process. From these insights, we synthesized a Self-Other Continuum represented by a sense of agency and facial identity. This continuum has implications in terms of the social and subjective aspects of interpersonal communication, which enables further scenario design and could complement findings from research on interactive devices for remote communication.

Where does one’s own face end?

A face carries two things at once: facial signals, such as expressions, gaze and mouth movements, and a facial identity that marks the boundary between self and other. Psychology has shown that this boundary can move. In the enfacement illusion, people come to feel another person’s face as their own when they see it touched in sync with their own face. But the technique takes time to induce and requires the body to stay still, so it leaves out conversation and the non-verbal signals people exchange. We asked where the boundary between self and other lies when two people keep talking and moving while their faces gradually turn into each other’s.

Blending two faces in real time

Morphing Identity separates each face into its static identity and its real-time signals, and recombines them. Both users’ faces are projected into the latent space of StyleGAN2, which yields 100 interpolated faces from one user to the other. A face reenactment model then animates the current step with the user’s own expressions and head pose, with an end-to-end latency of about 300 ms. The screen shows both faces side by side: one’s own face slowly turning into the partner’s, and the partner’s face turning into one’s own. Each user always moves the face on their side, so by the end each is moving a face that has become the partner’s, while the partner moves theirs.

Each user’s face is decomposed into a static facial identity and real-time facial signals; a sequence of faces morphed from one user to the other is recomposed with each user’s real-time movement — Each face is decomposed into a static facial identity and real-time facial signals. The facial identity is morphed step by step toward the partner’s, and each user moves the morphed face with their own expressions and head pose.

Thousands of judgments at public exhibitions

To include diverse relationships, from strangers to couples and families, we installed the system in two mirrored, semi-open booths at public exhibitions. Each participant held a foot switch for as long as the face on the screen still looked like their own. Morphing advanced at 2% per second until the faces were fully swapped, then returned. The gradual change makes it hard to see the face changing, so the switch measures recognition rather than the detection of a change. We collected 9,269 responses and analyzed 3,411 after excluding unreliable ones, measuring each boundary as a distance in the GAN latent space so that pairs of very different faces could be compared. We also interviewed 31 pairs (62 people: family members, couples, friends and others).

Two people sit in two mirrored white booths, each facing a screen — The two mirrored, semi-open booths. Each participant sees both faces side by side and can talk with the partner.

(a) Visitors queuing at the exhibition booths; (b) the sequence: entering the booth, face capture, consent, start of morphing, foot switch release at the self boundary, full swap and return — (a) The exhibition. (b) The sequence: each participant holds a foot switch while the face on the screen still looks like their own and releases it when it no longer does; the faces are fully swapped and then return.

A continuum between self and other

The participants did not flip from self to other at one point: their sense of facial identity moved along a continuum, from extended through joint to intertwined, while they kept a sense of agency over the face they were moving at every stage.

The first part of the continuum is a window of self-identification. Up to a morphing level of about 20%, participants still recognized the face as their own, even though it already contained part of their partner’s, and many did not notice the change: “I did not realize there were some changes until the middle.” Here facial identity is extended: one’s own face quietly takes in some of the other.

(a) Histogram of reported self-face boundaries with a peak around a latent distance of 18; (b) cumulative share of participants still reporting their own face, falling to half at about 23; (c, d) the same by gender and age — Where participants stopped recognizing the face as their own (N = 3411), measured as distance in the GAN latent space. (a) The boundaries peak at a distance of 18.2. (b) Half of the participants still saw their own face at 23.4, a morphing level of about 20%. (c, d) By recognized gender and age.

Beyond this window the faces became joint. Participants described them as new but still made of their own identity: “When I explicitly noticed it, the face already looked like someone other than me, but at the same time, it somehow felt familiar.” The mixed faces became a shared topic. Pairs talked about family resemblance and even future generations (“I could see the potential face of our future generation”).

At high morphing levels the identities were intertwined. Each participant moved a face that had become the partner’s, and some enjoyed it: “I could fully manipulate the face and its expression, which is completely different from my face.” At the same time, some still felt that the face their partner was moving was their own, and found this strange or embarrassing. One mother, morphed with her son, said: “Stop playing with my face!”

The Self-Other Continuum from low to high morphing level: extended, joint and intertwined facial identity, with arrows for sense of agency and facial identity between users and morphed faces, and an example scenario for each — The Self-Other Continuum. Sense of agency (SoA) stays with each user’s own movement, while facial identity (FI) is first extended, then shared, and finally crossed over to the partner.

Agency and identity, separated

Because the system separates facial identity from facial signals, the face one moves and the face one recognizes as oneself can belong to different people. In everyday life the two coincide; along the continuum they come apart, and at its far end they cross: “I am moving the partner’s face,” but “my partner is also moving my face.” No participant commented on a loss of control at any stage, so it was facial identity, not agency, that shifted.

The window of self-identification was also narrower than the 40–60% reported in enfacement studies with static faces. The paper points to three likely reasons: real-time movement gives more cues for telling faces apart, pairs here could be of different genders, and participants answered the stricter question of whether the face was “me”.

Designing with the continuum

Each region of the continuum suggests a different use. Within the extended region, a little of a partner’s identity could be blended into one’s face without being noticed, for example to increase closeness in a video call, since facial resemblance is known to increase trust and liking. A joint face could let two people act as one avatar and share both responsibility and achievement, such as a new employee and a senior instructor serving customers together. An intertwined face could let someone borrow a trusted partner’s face, with consent. Without consent, the same swap can violate a person’s facial identity, and because reactions to swapping ranged from enjoyment to disgust, the paper argues against a one-size-fits-all design.

Exhibitions

What is our identity? Morphing Identity system provides an experience where your face will be transformed into the face of another person. With leveraging the human perceptual characteristics of visual changes, computationally created perceptually smooth change is applied to the facial image in real-time. Our integrated system composites realistic real-time morphing images of users’ faces by performing face animation reenactment with interpolated projected latent codes of two faces in a latent space of a GAN (Generative Adversarial Network). This perceptually gradual face morphing composition allows us to change the face image without perceiving the visual change. By focusing on the fluidity of facial identity we have, we explore how the self that you think you are and the self that others think you are can be transformed.

私たちのアイデンティティとは何か。本作品は、自分の顔だと分かっている筈なのに、自分だった顔がいつの間にか他者の顔に変容していく体験を提供する。人間の変化知覚特性に基づいて、計算的に作り出された知覚的に滑らかな変化を顔映像に適応して、リアルタイムに自他の比率を徐々に変えながら顔映像を再合成することで、変化を知覚しない顔画像の変化を引き起こす。映像を介したコミュニケーションにおいて、個人を識別するための顔映像が持つアイデンティティの流動性に着目し、自分自身が思う自分と、他者が思う自分がどのように変容し得るかを探索する体験型の作品である。

Credit for exhibition projects

Concept and Research
– Shunichi Kasahara (Sony CSL)
System
– Shunichi Kasahara (Sony CSL)
– Kye Shimizu (Sony CSL)
– Santa Naruse (Sony CSL)
– Naoto Ienaga (Tsukuba University)
Technical Assistant
– Kazuma Takada (Sony CSL)
– Taku Tanichi (Sony CSL)
– Maki Sugimoto (Keio University)

Exhibition

Special Exhibition ‘You and Robots / What is it to be Human?’, 2022.3.18 – 8,31, The National Museum of Emerging Science and Innovation

Media Ambition Tokyo (MAT). 2021.5.12WED– 6.8TUE, Tokyo City View, 52F, Roppongi Hills Mori Tower

Sony Park Exhibition”Research for the Future of Humanity“. 2021.07.24-2021.08.10. Ginza Sony Park

Reference

Kye Shimizu, Santa Naruse, Jun Nishida, and Shunichi Kasahara. 2023. Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3544548.3580853

BibTeX

@inproceedings{10.1145/3544548.3580853,
  author    = {Shimizu, Kye and Naruse, Santa and Nishida, Jun and Kasahara, Shunichi},
  title     = {Morphing Identity: Exploring Self-Other Identity Continuum through Interpersonal Facial Morphing Experience},
  booktitle = {Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '23},
  year      = {2023},
  pages     = {1--15},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3544548.3580853}
}

Related projects: Face boundary in action, Control over self and others’ face, Human Latent Metrics