Stealth Updates of Visual Information by Leveraging Change Blindness and Computational Visual Morphing
Shunichi Kasahara and Kazuma Takada
ACM Transactions on Applied Perception (TAP) 2021, presented at ACM SAP 2021
Paper (ACM Digital Library) · PDF
Abstract
We present an approach for covert visual updates by leveraging change blindness with computationally generated morphed images. To clarify the design parameters for intentionally suppressing change detection with morphing visuals, we investigated the visual change detection in three temporal behaviors: visual blank, eye-blink, and step-sequential changes. The results showed a robust trend of change blindness with a blank of more than 33.3 ms and with eye-blink. Our sequential change study revealed that participants did not recognize changes until an average of 57 % morphing toward another face in small change steps. In addition, changes went unnoticed until the end of morphing in more than 10 % of all trials. Our findings should contribute to the design of covert visual updates without consuming users’ attention by leveraging change blindness with computational visual morphing.
Updating a display without drawing attention
Pervasive displays and wearable displays such as AR glasses can keep the information they show up to date. Every update, however, is a visible change, and a visible change draws attention away from whatever the user was doing. We asked how visual information could be updated without drawing attention.
Change blindness offers a way. People often fail to notice even obvious changes when the visual transient that accompanies a change is masked, for example by a brief blank screen, an eye movement or a blink. Masking alone does not hide large changes, which people can still detect by comparing what they see with their visual short-term memory. Our approach therefore generates a series of intermediate images computationally and applies a change in small steps, each hidden behind a brief interruption. To design such stealth updates, we needed to know how the size of each change and its timing affect whether people notice it.
Morphing faces, scenes and text
We generated morphing sequences for three kinds of visual information. Faces were morphed by interpolating between two points in the latent space of StyleGAN2, and other images (landscapes, animals, food and everyday objects) with BigGAN. Text cannot be interpolated pixel by pixel, so BERT replaced the nouns, adjectives and verbs of a short movie review, one word at a time, with words that fit the context. The change rate is the fraction of the whole morph applied in a single change. For faces, a change of 0.05 or 0.1 leaves two images looking like the same person; a change of 0.5 makes them look like different people.
Blanks and blinks
In the first study, 12 participants watched four images on a 240 Hz display. After a gray blank of up to 266.7 ms, or no blank at all, one of the images changed, and participants pressed the key for that image or reported that they had seen no change. Detection fell sharply as the blank lengthened and leveled off for blanks of 33.3 ms or longer in all three categories. For small changes (a change rate of 0.1), a blank of 16.7 ms, one frame on a 60 Hz display, was already enough, so a stealth update does not need a longer, more distracting interruption. For faces and other images, large changes (0.5) were still detected far more often than small ones; for text, the size of the change made no difference.
A blank of 33 ms is shorter than a typical human blink. In the second study, the same participants did the task while a camera running at 140 frames per second detected their spontaneous blinks, and the image changed during a blink. Small changes were detected in 10% of trials for faces and text and in 12% for other images, similar to the results with blanks of 33 ms or longer. Participants said they had not noticed when an image changed, even while they were concentrating on the task. A blink can hide a change as well as a blank can, without adding any visible interruption.
A sequence of small changes
In both studies, a single small change was detected in only about 10% of trials. The next question was how far a change could go when it was applied as a series of such steps. In the third study, 18 participants took part online. One of four images changed one step at a time during a 66.6 ms blank that occurred every 2.5 seconds, about the rate of natural blinks, in steps of 0.05, 0.1 or 0.5 of the whole morph. Participants were told to press a key as soon as they found which image was changing, so they were looking for the change. For faces and other images, a gradual mode also changed the image continuously over 25 or 50 seconds without blanks, at the same overall speed as the steps of 0.1 and 0.05. We measured a stealth score: how far the morph had progressed before the participant noticed it, where 1.0 means the change was never noticed.
Turning one face into another
Applied in small steps behind brief blanks, a change went unnoticed far beyond the size of any single step: participants did not notice a face turning into another person’s face until it had morphed 57% of the way on average, and in more than 10% of all trials they never noticed the change at all. On average, then, the face had moved further from the original than the change of 0.5 that already makes two faces look like different people, while participants were watching for exactly such a change.
The size of the steps mattered in different ways. For faces, large steps of 0.5 were noticed earlier (an average stealth score of 0.30), but steps of 0.05 did no better than steps of 0.1, probably because a smaller step needs more steps to reach the same change, and every step is another chance to notice. For text, where the size of a single change had made no difference in the first two studies, the trend reversed: the stealth score rose with the step size (0.42, 0.53 and 0.78), which suggests that the number of steps decided. For other images the step size made no consistent difference. Once participants had found a change in one food image, they knew what to look for in similar images, and some used the visual artifacts typical of the image generator as clues.
When the same change ran continuously without blanks, participants tended to notice it earlier, and they said the gradual mode was easier. Observers seem to be less sensitive to a change spread over blanks and blinks than to a gradual one.
Why large changes go unnoticed
A blank or a blink removes the visual transient that would otherwise draw attention to a change, so a person has to detect the change by comparing the current image with a memory of the earlier one. We think that each update also overwrites that memory. Earlier work has shown that people cannot hold two complete visual representations at once, one of the mechanisms behind change blindness, and in the third study all four images, including the changing one, stayed in view. The memory of the original face would then be updated along with the face, leaving participants without a cue to detect the change even when the total change was large.
These findings give designers parameters for covert updates: a blank of a single frame is enough for a small change, blinks can serve as natural blanks, and the size of each step has to be balanced against the number of steps. Designers can build on them to lay out information that is updated covertly, taking the user’s attention and gaze into account, so that displays such as AR glasses stay up to date without consuming the user’s attention.
Talk at ACM SAP 2021
Reference
Shunichi Kasahara and Kazuma Takada. 2021. Stealth Updates of Visual Information by Leveraging Change Blindness and Computational Visual Morphing. ACM Transactions on Applied Perception 18, 4, Article 23 (October 2021), 17 pages. https://doi.org/10.1145/3486581
BibTeX
@article{10.1145/3486581,
author = {Kasahara, Shunichi and Takada, Kazuma},
title = {Stealth Updates of Visual Information by Leveraging Change Blindness and Computational Visual Morphing},
journal = {ACM Transactions on Applied Perception},
volume = {18},
number = {4},
year = {2021},
articleno = {23},
pages = {1--17},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3486581}
}Figures from the paper (Kasahara and Takada, ACM TAP 2021). © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.