Human Latent Metrics

Human Latent Metrics: Perceptual and Cognitive Response Correlates to Distance in GAN Latent Space for Facial Images

Kye Shimizu, Naoto Ienaga, Kazuma Takada, Maki Sugimoto, and Shunichi Kasahara
ACM SAP 2022

Abstract

Generative adversarial networks (GANs) generate high-dimensional vector spaces (latent spaces) that can interchangeably represent vectors as images. Advancements have extended their ability to computationally generate images indistinguishable from real images such as faces, and more importantly, to manipulate images using their inherit vector values in the latent space. This interchangeability of latent vectors has the potential to calculate not only the distance in the latent space, but also the human perceptual and cognitive distance toward images, that is, how humans perceive and recognize images. However, it is still unclear how the distance in the latent space correlates with human perception and cognition. Our studies investigated the relationship between latent vectors and human perception or cognition through psycho-visual experiments that manipulates the latent vectors of face images. In the perception study, a change perception task was used to examine whether participants could perceive visual changes in face images before and after moving an arbitrary distance in the latent space. In the cognition study, a face recognition task was utilized to examine whether participants could recognize a face as the same, even after moving an arbitrary distance in the latent space. Our experiments show that the distance between face images in the latent space correlates with human perception and cognition for visual changes in face imagery, which can be modeled with a logistic function. By utilizing our methodology, it will be possible to interchangeably convert between the distance in the latent space and the metric of human perception and cognition, potentially leading to image processing that better reflects human perception and cognition.

Can a distance in latent space be a distance for people?

Generative adversarial networks such as StyleGAN2 represent a face image as a vector in a high-dimensional latent space, so a face can be edited by moving its vector: interpolating between two vectors morphs one face smoothly into another. How such a move relates to what people see had been described only qualitatively. Our earlier study of stealth updates used GAN morphing to change images without people noticing, but reported its results as percentages of the morph rather than as distances in the latent space. We asked whether the distance a face travels in the latent space corresponds to how people perceive and recognize the change, closely enough to be described by a simple model.

Two tasks on the same morphed faces

We generated faces with StyleGAN2 trained on the FFHQ dataset, removed unsuitable images, filled the background with uniform gray, and projected each image back into the latent space. For 250 pairs of faces we generated 31 interpolated images, so that every image corresponds exactly to a latent vector and every pair of images to a distance Δw.

Pipeline from seed to latent vector w, image selection with rejected and accepted faces, gray background filling, projection into an 18 × 512 dimensional latent space, and interpolated face sequences with their distances Δw — Generating the stimuli: faces generated with StyleGAN2 are filtered, given a gray background and projected back into the latent space; interpolating between two latent vectors gives a morphing sequence whose length is the distance Δw.

The same images were used in two online experiments. In the change perception task (98 participants), four faces were shown, then a 66 ms blank, then the same four faces with one of them moved by Δw; participants pressed the key for the face that had changed, or a key for no change. The blank suppresses motion cues, as in change blindness research, so the task measures whether the change itself is perceived. In the face recognition task (152 participants), faces were shown one after another, and a target face reappeared one or two faces later, moved by Δw; participants pressed a key when they recognized a face they had seen. After excluding inattentive participants, 2,560 responses from 88 participants and 2,760 responses from 138 participants were analysed.

Perceptual and cognitive distance in latent space

How often people noticed a change, and how often they no longer recognized the face as the same, both rose with the distance travelled in the latent space along a logistic curve, so a latent distance can be converted into a perceptual and a cognitive distance. The logistic models fit closely, with McFadden’s R² of 0.92 for both change perception and face recognition. Because both tasks used images from the same latent space, a single distance Δw predicts both how likely a change is to be noticed and how likely the face is to be taken for one already seen. This is the sense of the title: distances in a generative model’s latent space can serve as metrics of human perception and cognition, and the two can be converted into each other.

Two plots against Δw: change perception rate rising from about 0.1 to over 0.85, and face recognition rate (reversed axis) falling from about 0.85 to under 0.2, each with individual responses, binned averages and a logistic curve — Responses in the change perception task (left) and the face recognition task (right) against Δw. Dots are averages over bins of Δw and the curves are logistic regressions (McFadden’s R² = 0.92 for both). The recognition axis is reversed so that both curves rise with distance.

Perception and recognition depend on different parts of the vector

The latent vector has 18 layers of 512 dimensions, and different layers carry different visual information: in the paper’s material, the coarse layers (1–4) change shape and posture, the middle layers (5–8) change physiognomy, and the fine layers (9–18) change color and texture.

Two examples of morphing from one face to another with all layers, only coarse layers, only middle layers, or only fine layers interpolated — Morphing only some layers of the latent vector. Coarse layers change shape and posture, middle layers physiognomy, and fine layers color and texture.

Fitting the model to each layer separately showed different patterns for the two tasks. Change perception was generally better explained by the lower layers, which produce the larger visual changes. Face recognition was explained less well by the coarse layers than by the middle ones: whether a face is recognized as the same depends less on posture and more on physiognomy. The paper reads this as one latent space accounting for two different visual processes, perception and cognition.

Line chart of R² for each of the 18 layers, for change perception (red) and face recognition (blue); face recognition is low at layer 1 and both drop sharply at layer 17 — How well the distance in each of the 18 layers predicts change perception (CP, red) and face recognition (FR, blue), as R² of the logistic regression.

Editing images with people in mind

With such a model, an image edited in the latent space can be designed for how others will perceive it. A face could be changed in age or gender while still being recognized as the same person a chosen proportion of the time, for example 60%, or a person’s true facial identity could be masked while the face stays recognizable. The paper presents the procedure as a general way to relate a learned latent space to human perception and cognition, beyond StyleGAN2 and beyond faces, and notes that the recognition task here measured short-term memory only.

Reference

Kye Shimizu, Naoto Ienaga, Kazuma Takada, Maki Sugimoto, and Shunichi Kasahara. 2022. Human Latent Metrics: Perceptual and Cognitive Response Correlates to Distance in GAN Latent Space for Facial Images. In ACM Symposium on Applied Perception 2022 (SAP ’22). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3548814.3551460

BibTeX

@inproceedings{10.1145/3548814.3551460,
  author    = {Shimizu, Kye and Ienaga, Naoto and Takada, Kazuma and Sugimoto, Maki and Kasahara, Shunichi},
  title     = {Human Latent Metrics: Perceptual and Cognitive Response Correlates to Distance in GAN Latent Space for Facial Images},
  booktitle = {ACM Symposium on Applied Perception 2022},
  series    = {SAP '22},
  year      = {2022},
  pages     = {1--10},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3548814.3551460}
}

News: Full paper accepted to ACM SAP 2022
Related projects: Stealth Updates, Morphing Identity, Face boundary in action