Behind the Game

Behind The Game: Implicit Spatio-Temporal Intervention in Inter-personal Remote Physical Interactions on Playing Air Hockey

Azumi Maekawa, Hiroto Saito, Narin Okazaki, Shunichi Kasahara, and Masahiko Inami
ACM SIGGRAPH 2021 Emerging Technologies

The collaboration project with Azumi Maekawa, Hiroto Saito, Narin Okazaki and Masahiko Inami (RCAST, The University of Tokyo)

Paper (ACM Digital Library) · PDF

Abstract

When playing inter-personal sports games remotely, the time lag between user actions and feedback decreases the user’s performance and sense of agency. While computational assistance can improve performance, naive intervention independent of the context also compromises the user’s sense of agency. We propose a context-aware assistance method that retrieves both user performance and sense of agency, and we demonstrate the method using air hockey (a two-dimensional physical game) as a testbed. Our system includes a 2D plotter-like machine that controls the striker on half of the table surface, and a web application interface that enables manipulation of the striker from a remote location. Using our system, a remote player can play against a physical opponent from anywhere through a web browser. We designed the striker control assistance based on the context by computationally predicting the puck’s trajectory using a real-time captured video image. With this assistance, the remote player exhibits an improved performance without compromising their sense of agency, and both players can experience the excitement of the game.

Playing air hockey through a time lag

Playing a physical sport against someone in another place is hard because of the time lag between a player’s action and its feedback. The lag slows the remote player’s reactions, makes their control less accurate and weakens their sense of agency, the feeling of controlling one’s own actions. The remote player falls behind the physical one, and the unfair game takes away both players’ motivation. A computer could step in to improve the remote player’s performance, but an intervention that ignores what the player intends may also take away their sense of agency, and in a game between people that intention keeps changing with the opponent’s moves, the player’s own moves and the state of the table.

We built remote air hockey to explore this. On one half of the table, a 2D plotter-like machine moves a striker, driven by two DC motors through belts and pulleys so that it can keep up with a player’s arm. The remote player watches a live video of the table in a web browser and moves an indicator for the striker with a mouse or touchpad, against a physical player on the other side. The video reaches the remote player, and the player’s input reaches the machine, each with a delay of tens to hundreds of milliseconds.

Diagram of the remote player at a display and the local player at the air hockey table with the machine and a camera, with the delay of each step labeled in milliseconds — The remote player (left) controls the machine on the local table (right) over the network. Every step between the two sides adds a delay, from a few to hundreds of milliseconds.

Assistance that follows the context of play

Instead of correcting the striker all the time, the system steps in only at the two moments that decide air hockey, defense and offense, and only on the striker’s horizontal position; because the correction helps the player hit the puck as they meant to, the remote player plays better without losing the sense of agency. In defense, players in our preliminary experiments often felt more agency than when they moved the striker by themselves, because the striker’s motion appeared to match their action.

Watching people play, we found that the game divides into defense, when the player blocks or returns the opponent’s shot, and offense, when the player shoots the puck into the opponent’s side. In both, what matters most is horizontal accuracy: without it, the player misses the puck or lets the opponent’s shot through. When the system detects one of these contexts, it takes over the horizontal position for a moment, while the player keeps the vertical movement and still decides whether to shoot or keep the puck. After the hit, the assistance fades out gradually until the striker is entirely back under the player’s control.

Two video frames of the table during defense, with the puck, the striker and the user input marked, above a plot of the assistance rate jumping to 1 at the trigger, staying until the hit and then decaying to 0 — Assistance during defense. (a) When the system detects the context, the assistance turns on at once. (b) After the striker hits the puck, the assistance rate decays gradually until the player has full control again.

The two contexts are detected from the puck and the striker in the camera image. Defense starts when the puck comes toward the remote player above a certain speed; the system then moves the striker sideways to where the puck’s predicted path crosses the striker’s line. This correction starts without waiting for the player’s input, because a reaction to the delayed video would come too late. Offense starts when the puck stays slow on the remote player’s side and the player moves the striker to hit it; the system then lines the striker up with the puck.

Two camera views of the table: in defense the striker is moved sideways to meet the puck’s predicted path; in offense the user moves the striker forward and the assistance shifts it sideways onto the puck — (a) Defense: the striker moves to the contact point estimated from the puck’s path. (b) Offense: the striker moves to the position of the puck.

When the machine does more

Assistance that did more did not keep the same feeling. A prototype that also corrected the vertical movement, aiming at a contact point in front of the striker, shot the puck with more power than the horizontal assistance, but some test users reported a lower sense of agency. The intervention may have produced movements and timings that differed from the players’ intentions, possibly because the context was estimated inaccurately; switching between horizontal and two-dimensional assistance according to the player’s input might add the extra power without that cost. Optional haptic feedback, for the collision with the puck and a spring force toward the corrected position, may help players become proficient, but it can also make the intervention apparent.

The demonstration for SIGGRAPH 2021 Emerging Technologies was planned so that attendees could join from a web browser and play with their mouse against a physical opponent. We plan a detailed user study to find how to set the assistance so that it maximizes the sense of agency, and to see how people’s behavior changes as the intervention increases or decreases and as they learn to move together with the machine.

Reference

Azumi Maekawa, Hiroto Saito, Narin Okazaki, Shunichi Kasahara, and Masahiko Inami. 2021. Behind The Game: Implicit Spatio-Temporal Intervention in Inter-personal Remote Physical Interactions on Playing Air Hockey. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies (SIGGRAPH ’21). Association for Computing Machinery, New York, NY, USA, 1–4. https://doi.org/10.1145/3450550.3465348

BibTeX

@inproceedings{10.1145/3450550.3465348,
  author    = {Maekawa, Azumi and Saito, Hiroto and Okazaki, Narin and Kasahara, Shunichi and Inami, Masahiko},
  title     = {Behind The Game: Implicit Spatio-Temporal Intervention in Inter-personal Remote Physical Interactions on Playing Air Hockey},
  booktitle = {Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies},
  series    = {SIGGRAPH '21},
  year      = {2021},
  pages     = {1--4},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3450550.3465348}
}

Related projects: The Tight Game, Preemptive Action