Malleable Embodiment

Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body

Shunichi Kasahara, Keina Konno, Richi Owaki, Tsubasa Nishi, Akiko Takeshita, Takayuki Ito, Shoko Kasuga, and Junichi Ushiba
ACM CHI 2017

This research is part of “Superception” (super + perception), a research and engineering framework that Shunichi Kasahara is building at Sony CSL to transform and expand human perception and recognition. It was carried out with YCAM (Yamaguchi Center for Arts and Media), studying human embodiment through virtual body expressions in computer graphics.

Paper (ACM Digital Library) · PDF

Abstract

We hypothesize that replacing the visual perception of one’s body with a spatial-temporal deformed state would change sensations associated with the body. We developed a system that captures full-body movement and generates estimated past and future body movement by deformation. With a head mounted display, people could see their bodies as slightly deformed. We then investigated 1) how human movement is physically changed, and 2) how humans feel about the change in physical and emotional views of the body due to virtual body deformation. Our results show that spatial-temporal deformation of a virtual body actually changes the sense of body as well as physical movement. For instance, a body image generated at approximately 25-100 ms in the future induced a “lighter weight” sensation. On the basis of our findings, we discuss the design implication of computational control for the physical and emotional sense of body.

Can the sense of one’s body be made malleable?

The feeling that “this is my body”, body ownership, enables people to adapt to an environment or a tool and to extend their physical sensations. It is not tied to the body itself. In the rubber hand illusion, simultaneous visual and tactile stimulation makes people perceive a rubber hand as their own, and in immersive virtual reality people can come to feel a computer-generated body as theirs. Earlier studies changed the posture, appearance or size of such virtual bodies and found effects on how people feel.

We asked whether the movement of a virtual body could change the sense of body in the same way: if people see their own body move slightly ahead of or behind their actual movement, do the sensations associated with their body change? Two questions guided the work: how a deformed virtual body changes people’s physical movement, and how people feel, physically and emotionally, about their body when they see it deformed. The design space of possible deformations is very large, so we required that the virtual body keep the structure of a human body and remain one’s own.

A virtual body shifted in time

The system captures full-body movement with a motion-capture suit of 37 markers and eight infrared cameras, and turns it into a skeleton of 23 joints at 120 frames per second. From each joint’s recent velocity, it estimates where the joint will be slightly in the future, or was slightly in the past, and renders this deformed body from a first-person view in a head-mounted display, together with a virtual floor, boxes and a virtual mirror.

A dancer wearing a head-mounted display in front of a screen showing skeletons; the motion capture cameras feed a module that generates the spatial-temporal deformed skeleton and renders the first-person view — Motion capture records the full body; the system generates a skeleton shifted in time and renders it from the user’s first-person view in a head-mounted display.

A single parameter T sets the shift. At 60 frames per second, T = 3.0 shows each joint about 50 ms ahead of the system’s baseline, and T = −3.0 about 50 ms behind; the baseline itself lagged the physical movement by about 70 ms. The virtual body was drawn as a simple skeleton of boxes and lines. In pilot studies this was enough for people to feel it as their own body, while a volumetric surface on top of the skeleton sometimes weakened the illusion through small mismatches with the real body and clothing.

(a) Diagram of skeleton joints: each joint position is moved along its smoothed velocity by T to give the deformed position. (b) Three frames of a person waving both arms clockwise, with the original skeleton in yellow and the deformed skeleton in blue slightly ahead — (a) Each joint is moved along its smoothed velocity by the parameter T. (b) A person waving both arms clockwise: the yellow skeleton shows the current body (T = 0) and the blue one the body estimated about 50 ms ahead (T = 3.0).

Reaching with a shifted body

Seventeen participants sat wearing the head-mounted display and repeatedly reached with the right hand for a virtual box. Every ten trials, the deformation changed without announcement, within a range of T from −3.0 to 3.0, where pilot tests had shown that people did not notice it.

The further ahead the virtual body was shifted, the shorter participants reached. The reaching distance fell as T rose for all but one of the 17 participants, and in some of them it changed by more than 10 cm between the two ends of the range. Participants generally did not notice the deformation or the change in their movement. When they missed the target, they blamed themselves: “I thought that I was too tired since I could not reach the target.”

Regression lines of reaching distance against the deformation parameter T from −3.0 to 3.0 for each of the 17 participants, all but one sloping downward, with correlation coefficients from −0.60 to 0.04 — Linear regression between the deformation parameter T and the reaching distance for each participant, with the correlation coefficient in brackets. For all but one participant, the reach got shorter as the virtual body was shifted further ahead.

A workshop with people who know their bodies

To learn how the deformation feels, we held a workshop with 12 people who are highly aware of their bodies through their work or practice, including a professional ballet dancer, a contemporary dancer, a dance instructor, a jazz dancer, a kung-fu master and a triathlete. In the first session, participants moved freely, for example dancing ballet or practicing kung-fu, while we changed the deformation, and they described whether they felt the virtual body as their own and how they felt, without knowing what the system was doing. In the second session we explained the deformation and explored its effects together with them.

Lighter in the future, heavier in the past

As long as participants still felt the virtual body was their own, seeing it slightly ahead in time made their body feel lighter, nimbler and more vitalized, and seeing it slightly behind made it feel heavier, duller and diminished, although their actual body had not changed. This symmetric profile was clearest with small deformations, up to about 30 ms ahead or behind (T between −2 and 2). Participants described these changes as changes in their own body: “I got a lighter body”, “I feel my body is a bit heavy”, “Now my body is nimble as I wanted it to be”. Emotional statements came with them, such as “Hey, did you hack something? I am now more vitalized.”

Map of participants’ statements along the temporal deformation from −80 ms to +80 ms: near zero, imperceptible; toward +30 ms, lighter, fast and vitalized; toward −30 ms, heavier, slow and exhausting; beyond ±80 ms, discriminated, with statements such as “It looks like seaweed” and “It looks like comic character” — Statements collected in the workshop, arranged by the temporal deformation from 80 ms behind to 80 ms ahead. Shifts toward the future brought statements such as “lighter weight” and “vitalized”, shifts toward the past “heavier” and “exhausting”.

The statements fell into three states, arranged roughly symmetrically around the undeformed body. With the smallest deformations, participants noticed no change and kept ownership of the virtual body (imperceptible). With somewhat larger ones, they perceived a change in their body and still felt the virtual body as their own (perceptible change with ownership); the sensations of weight and vitality appeared here. With large deformations, they distinguished the virtual body from their own and ownership broke down (discriminated): “It’s like seaweed, now my arm looks like seaweed”, “It looks like a spring attached to my arm.” The boundaries between the states varied with each person’s sensitivity.

Changes in physical sensation were mentioned only within the range where participants kept ownership of the virtual body. Once ownership broke down, participants no longer described changes in their own body, but they still felt connected to the virtual body and changed how they moved. One professional dancer said, “I’m actually happy to use this virtual spring attached to my body. By playing with it, I got inspiration for body movement.”

Intrinsic modulation and extrinsic inspiration

The two ways people interpreted the conflict between what they saw and what they felt define two design spaces for deforming a virtual body. While body ownership is preserved, people unconsciously read the conflict as a change in their own body: intrinsic modulation of the sense of body, such as its weight, size, speed and condition. Its effect has to stay small to keep ownership, but it works without people noticing. When ownership is lost but a sense of connection with the virtual body remains, people treat the virtual body as an object attached to them or an extension of their body that inspires their movement: extrinsic inspiration. Here the virtual body can be expressed much more freely, and the emotional effect remains; a ballet dancer said of the smoothed body, “I feel comfortable and feel elegant to see the smoothed version of my body. It’s really elegant, which is what I want to do in my ballet expression Adagio.”

Diagram of the design space along the deformation of the virtual body: intrinsic modulation near the original body, under body ownership, with physical sensations such as weight, size, speed and emotional ones such as vitality; extrinsic inspiration further out, under a sense of connection, with interactive tool, body attached object, extension of body and personality of the virtual body — Design space of virtual body deformation. Near the original body, where body ownership holds, deformation modulates physical and emotional sensations from within; further out, where only a sense of connection remains, the virtual body inspires movement from outside.

In a paired session, three pairs of participants shared the virtual space and saw each other’s bodies. When both bodies were shifted forward, each felt the partner’s movement as lighter and livelier as well as their own. When the two saw different deformations, one forward and one backward, they believed they were in the same condition as long as they held hands. Giving different people different deformations could thus create a kind of parallel reality in how they sense their bodies.

Where the illusion breaks

Ownership did not hold in every situation. With small movements such as waving, a forward-shifted body could look as if it moved before the person did: “This [the virtual body] looks like it is reading my mind and goes there before I go.” Large forward shifts produced an overshoot when the hand stopped (“I feel it [the virtual body] is bouncy”), an artifact of the simple estimate of future movement. People with mastery of a movement, such as kung-fu forms or swimming strokes, noticed small discrepancies more easily, and touching one’s own body or another person, for example by clapping, revealed the mismatch between sight and touch. Why a forward shift feels lighter and a backward shift heavier remains to be explained.

Toward computational control of the sense of body

These results open the possibility of controlling by computation how a person’s body is perceived, along with the feelings and emotions that come with it. Gradual, scheduled changes in the intrinsic modulation space could adjust the physical condition of athletes and dancers or support physical and mental rehabilitation, and a game could make a player actually feel exhausted when hit and revitalized by virtual medicine. The extrinsic inspiration space suits dance creation, sports training and therapy, where a virtual body or object connected to one’s own can guide or inspire movement. A mixed reality version with an optical see-through display could ground the deformation in the real body.

Videos from CHI 2017

Presentation at CHI 2017.

Video preview for CHI 2017.

Reference

Shunichi Kasahara, Keina Konno, Richi Owaki, Tsubasa Nishi, Akiko Takeshita, Takayuki Ito, Shoko Kasuga, and Junichi Ushiba. 2017. Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). Association for Computing Machinery, New York, NY, USA, 6438–6448. https://doi.org/10.1145/3025453.3025962

BibTeX

@inproceedings{10.1145/3025453.3025962,
  author    = {Kasahara, Shunichi and Konno, Keina and Owaki, Richi and Nishi, Tsubasa and Takeshita, Akiko and Ito, Takayuki and Kasuga, Shoko and Ushiba, Junichi},
  title     = {Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body},
  booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '17},
  year      = {2017},
  pages     = {6438--6448},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3025453.3025962}
}

News: Superception Research Update 2021
Related projects: Parallel Eyes, Morphing Identity