Whose touch is this?

Whose Touch is This?: Understanding the Agency Trade-Off Between User-Driven Touch vs. Computer-Driven Touch

Daisuke Tajima, Jun Nishida, Pedro Lopes, and Shunichi Kasahara
ACM TOCHI 2022

This is a collaborative research project with Sony CSL, Superception Group and University of Chicago, Human Computer Integration Lab

Paper (ACM Digital Library) · PDF

Abstract

Force-feedback interfaces actuate the user’s to touch involuntarily (using exoskeletons or electrical muscle stimulation); we refer to this as computer-driven touch. Unfortunately, forcing users to touch causes a loss of their sense of agency. While we found that delaying the timing of computer-driven touch preserves agency, they only considered the naive case when user-driven touch is aligned with computer-driven touch. We argue this is unlikely as it assumes we can perfectly predict user-touches. But, what about all the remainder situations: when the haptics forces the user into an outcome they did not intend or assists the user in an outcome they would not achieve alone? We unveil, via an experiment, what happens in these novel situations. From our findings, we synthesize a framework that enables researchers of digital-touch systems to trade-off between haptic-assistance vs. sense-of-agency.

When the computer moves your finger

Haptic interfaces such as exoskeletons and electrical muscle stimulation (EMS) can move a user’s body so that it touches by itself, which the paper calls computer-driven touch. Such touches can speed up reactions, share a pose or a rhythm with a remote person, and convey emotions. To help, however, the device often has to move the user before they would have moved themselves, and being moved this way costs users their sense of agency, the feeling that they are the ones acting.

Our earlier project, Preemptive Action, showed that timing helps: delaying the EMS so that it fires closer to the user’s own movement preserves some agency while still speeding the user up. That study, though, only covered the case in which the computer touches exactly where the user intended. Predicting a user’s intention that precisely is rarely possible. What happens when the computer drives the user to an answer they did not choose, or to a correct answer they would have missed? Whose touch is it then?

A two-handed Stroop test with EMS

We built a variation of the Stroop test, a classic task that makes people hesitate between two answers. An instruction such as “touch red color” appears for 300 ms, followed by two words printed in mismatched colors, and the participant answers by tapping the left or right side of a touchscreen with the ring finger of that hand. In a pre-study without EMS, the 15 eligible participants took 516 ms on average to answer and were correct 77% of the time, so they made a mistake about once every five touches.

In the main study, EMS on both forearms could also tap for the participant, either on the correct answer (assistive touch) or on the wrong one (adversarial touch), at one of seven timings between 100 and 600 ms after the words appeared. After every trial the participant rated their agency from 1 (“I did not do it”) to 7 (“I did it”), giving 1,680 rated trials from 15 participants. Because the participant also chose an answer, the user’s touch and the computer’s touch could agree or disagree, and the outcome could be a success or a failure.

Whose movement is it?

Whether participants felt the movement was their own, their sense of agency, depended not only on when the EMS moved their finger but on whether it moved toward the answer they had chosen: when the two agreed, agency changed gradually with the timing; when they disagreed, it split sharply, decided by who moved first.

When the user and the EMS chose the same answer, agency rose smoothly the later the EMS fired, as in Preemptive Action. When they chose different answers, there was no gradual curve. Agency fell into two distinct levels, high when the participant touched before the EMS and low when the EMS touched first, almost like a step; whether the EMS was assistive or adversarial did not change this. With conflicting movements, people could tell much more easily who had acted.

Success and failure bias agency

The outcome biased the sense of agency: for the same movement at the same timing, participants felt more agency when the result was correct than when it was wrong. The bias appeared both when the user and the EMS agreed and when they disagreed.

It also changes how early a system can move the user. When the two agreed, the halfway point of agency was reached with the EMS firing about 50 ms earlier for correct outcomes than for wrong ones. Even including the roughly 100 ms latency of the apparatus, the movements at that point (about 430 and 480 ms) came before the participants’ unassisted reaction time of about 500 ms, so a system that helps users succeed can speed them up further while keeping part of their sense of agency.

Why outcomes and conflicts shape agency

The bias toward good outcomes resembles a well-known tendency to credit successes to ourselves and failures to outside causes. What is new is that it appeared even when the body had been moved by EMS. One explanation the paper discusses is postdiction: looking back on an action, we are more likely to claim it when it turned out well.

The sharp split in conflicting touches may come from cues that are easier to notice when the two touches differ. EMS causes a tingling sensation, and it contracts a muscle the user did not intend to move; on the hand the user meant to move, both are dampened by sensory attenuation, the weaker perception of the consequences of one’s own actions. When both hands move but only one was meant to, the user can tell who caused the touch. The paper notes that further experiments are needed to pin down these causes.

A trade-off between agency and assistance

From these results, the paper builds a matrix of four situations, defined by whether the user’s touch and the computer’s touch are correct, with the agency curve that applies in each.

• Joint success: the system speeds up a touch the user would have made anyway, and timing can preserve part of the user’s agency. This suits tasks where intention is easy to predict, such as catching a falling pen or pressing a camera shutter at the right moment.
• Forced success: the system corrects the user, at the cost of agency. This is worth it when safety comes first, for example pulling a hand away from a hot object or steering a walker away from an obstacle.
• Forced failure: the case to avoid. If the system’s confidence in its prediction is low, it should not assist at all.
• Joint failure: speeding a user toward a mistake makes the loss of agency worse through the outcome bias. The paper also sees an opportunity here: an interface that takes the blame for an imminent failure might help keep users motivated in demanding training.

A designer who wants to preserve agency should aim for agreement between the user and the system, using the recognizer’s confidence to regulate how much to assist. A designer who needs correct outcomes, as in emergencies, can accept forced success and trade away agency.

Reference

Daisuke Tajima, Jun Nishida, Pedro Lopes, and Shunichi Kasahara. 2022. Whose Touch is This?: Understanding the Agency Trade-Off Between User-Driven Touch vs. Computer-Driven Touch. ACM Transactions on Computer-Human Interaction 29, 3, Article 24 (January 2022), 27 pages. https://doi.org/10.1145/3489608

@article{10.1145/3489608,
  author    = {Tajima, Daisuke and Nishida, Jun and Lopes, Pedro and Kasahara, Shunichi},
  title     = {Whose Touch is This?: Understanding the Agency Trade-Off Between User-Driven Touch vs. Computer-Driven Touch},
  journal   = {ACM Transactions on Computer-Human Interaction},
  volume    = {29},
  number    = {3},
  year      = {2022},
  articleno = {24},
  pages     = {1--27},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3489608}
}

Figures from the paper (Tajima et al., ACM TOCHI 2022). © 2022 Copyright held by the owner/author(s).

• News: Our new paper from ACM TOCHI, CHI 2022 talk at the TOCHI journal session
• Related projects: Preemptive Action, EMS Speeds Up, Faster Than Muscle