ShadowClones

ShadowClones: an Interface to Maintain a Multiple Sense of Body-space Coordination in Multiple Visual Perspectives

Kazuma Takada, Nanako Kumasaki, Tom Froese, Kazuhisa Shibata, Jun Nishida, and Shunichi Kasahara
ACM Augmented Humans 2023

Paper (ACM Digital Library) · PDF

Abstract

We propose ShadowClones, an interface that supports interactions in which a single user can interact with multiple bodies in multiple spaces. Recent teleoperation technologies have allowed a user controlling multiple objects simultaneously, but at the same time, it also exhibited a significant challenge, which can be attributed to the high cognitive load caused by switching and recogning various spaces/perspectives repeatedly and instantly. To tackle this challenge, by taking advantage of pre-attentive visual cues for users’ simultaneous information processing, we designed and evaluated a new user interface, called Shadow Clones, that projects self-body information in unattended areas for increasing the awareness of body-space relationships and allowing users to seamlessly switch across different visual perspectives from avatars or remote robots. We then explored the proposed approach through a simple visual reaching task with a performance evaluation in terms of task completion time and success rate. The results showed superior performance when compared with a condition that presents no projections of users’ body movements in unattended areas. We conclude by discussing possible mechanisms of this enhancement as well as two potential scenarios using the shadow clones approach, including new entertainment content for virtual reality e-sports and multiple robot teleoperation such as in a construction site or a disaster site, without compromising operational performance.

Switching between many bodies

Networked systems now let one person operate several avatars or remote robots at once, each seen through its own view in a split-screen layout. Doing so means switching between views again and again, and every switch has a cost: the user has to make sense of the new view’s spatial coordinates and of where their own body, a hand or a cursor, is in it. We asked how a user could switch between bodies in different spaces repeatedly and instantly without this cognitive load.

Our approach builds on pre-attentive processing. People can keep track of several moving objects across the visual field without attending to each of them. If the user’s own movement were visible in every view, including the views they are not looking at, they might keep a sense of where their body is in each space and be ready to act as soon as they switch.

Shadows of one’s own movement

ShadowClones shows the user’s movement in all views at once. The user selects a view by looking at it, and that view holds the “body” cursor that the user actually controls. Every other view shows a dimmed “shadow” cursor that follows the same mouse movement in that view’s own coordinates but cannot act.

A reaching task across four views

In a user study, participants reached with a mouse for dot targets that appeared one after another in four views, each showing a randomly rotated plane, and had to reach each target within 1,000 ms. A new target appeared every 600 ms or 800 ms in another view, so they had to switch views for nearly every reach. In both conditions they switched views by gaze; the only difference was whether shadow cursors were shown in the views they were not looking at. The task used a cursor rather than a virtual body, the simplest setup, to rule out factors such as muscle fatigue and the appearance of the body.

Keeping a sense of the body in every space

Shadows of their own movement in the views they were not looking at let participants switch to another space and act there faster: the time to reach a target, which almost always meant switching views, was 57 ms shorter in median with ShadowClones than without the shadows. They also reached far more targets in time: the median share of targets reached in time rose from 41% to 73% when a new target appeared every 600 ms, and from 74% to 87% at 800 ms. Participants switched views more often as well. The paper reads this as the shadows maintaining a sense of body-space coordination in several spaces at once.

Why the shadows help

Each reach took several steps: noticing a new target, moving the gaze to it, switching the view, and adjusting the hand movement to the new view’s coordinates. Gaze-based switching handles the view switch with little cognitive effort. The paper also ruled out the system’s own timing as the source of the gain: the display followed an eye movement within about 20 ms, and a saccade between views took about 60 ms, so the 57 ms could not be explained by system delay alone. What the shadows add is the last step. Because the cursor’s position is already visible in every view, the user can grasp the relation between their movement and each space before looking at it, through an implicit process rather than a deliberate one.

Earlier multi-cursor techniques kept several cursors within a single coordinate system. ShadowClones keeps a separate body-space coordination for each space, which is what parallel control of bodies in different places requires.

Many bodies in many places

The same approach could let one user control the virtual hands of characters in several game worlds at once, as a new form of virtual reality e-sports, or let one operator run several construction machines or robots at a construction or disaster site from multiple perspectives, without losing operational performance.

Reference

Kazuma Takada, Nanako Kumasaki, Tom Froese, Kazuhisa Shibata, Jun Nishida, and Shunichi Kasahara. 2023. ShadowClones: an Interface to Maintain a Multiple Sense of Body-space Coordination in Multiple Visual Perspectives. In Proceedings of the Augmented Humans International Conference 2023 (AHs ’23). Association for Computing Machinery, New York, NY, USA, 171–178. https://doi.org/10.1145/3582700.3582706

@inproceedings{10.1145/3582700.3582706,
  author    = {Takada, Kazuma and Kumasaki, Nanako and Froese, Tom and Shibata, Kazuhisa and Nishida, Jun and Kasahara, Shunichi},
  title     = {ShadowClones: an Interface to Maintain a Multiple Sense of Body-space Coordination in Multiple Visual Perspectives},
  booktitle = {Proceedings of the Augmented Humans International Conference 2023},
  series    = {AHs '23},
  year      = {2023},
  pages     = {171--178},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3582700.3582706}
}

Figures from the paper (Takada et al., AHs ’23). © 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.

• Related projects: Parallel Ping-Pong, Parallel Adaptation