Parallel Eyes

Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto
ACM CHI 2016 · ACM SIGGRAPH 2016 VR Village

本プロジェクトは、YCAM (山口情報芸術センター)との共同研究です。
Research Collaboration with YCAM (The Yamaguchi Center for Arts and Media).

Paper (ACM Digital Library) · PDF

Abstract

Our research explores how humans can understand and develop viewing behaviors with mutual paralleled first person view sharing in which a person can see others’ first person video perspectives as well as their own perspective in realtime. We developed a paralleled first person view sharing system which consists of multiple video see-through head mounted displays and an embedded eye tracking system. With this system, four persons can see four shared first person videos of each other. We then conducted workshop based research with two activities, drawing pictures and playing a simple chasing game with our view sharing system. Our results show that 1) people can complement each other’s memory and decisions and 2) people can develop their viewing behaviors to understand their own physical embodiment and spatial relationship with others in complex situations. Our findings about patterns of viewing behavior and design implications will contribute to building design experience in paralleled view sharing applications.

Seeing through each other’s eyes

What if we could connect our sight directly and share it with other people? Could it extend our embodiment, complement our memory or extend our perception? As small wearable cameras become common, first person video is becoming a way to share one’s experience, but applications that stream it usually give people fixed roles: one person shoots with a head-mounted camera, and the others watch on a screen. Earlier work had explored two people exchanging first person video, yet little was known about configurations in which several people exchange their perspectives at once. We asked whether people can perceive and understand several perspectives at the same time, their own and those of others, and how they come to use them.

Four perspectives in one head-mounted display

Parallel Eyes is a technology probe for this question. Each user wears a video see-through head-mounted display made from a smartphone in a Google Cardboard viewer, with a conversion lens that widens the camera’s view to about 100 degrees. Four smartphones stream their camera video to each other over Wi-Fi, and each screen shows four first person videos in a split layout: the wearer’s own and those of the three others. The video runs at 20 to 30 frames per second, with the resolution adjusted to the available bandwidth.

The Parallel Eyes goggles from outside and inside, and a person wearing them with a small computer and battery on a belt — The Parallel Eyes goggles, a smartphone-based video see-through head-mounted display (left and center), worn with a small computer and a battery on the body (right).

A small infrared camera and LED inside the goggles capture one of the wearer’s eyes, and a computer detects the pupil to tell which of the four views the wearer is looking at. The wearers never see these traces in their goggles. The traces are recorded with the four videos, so that researchers, the audience and the participants can look back on what people did consciously and what happened without their noticing.

An operator watching the eye images of four participants on a screen, and a diagram of the goggles with a small eye camera beside the smartphone — The operator’s screen with the eye images of four participants (left), and the eye camera built into the goggles (right).

Workshops with drawing and a game of tag

Rather than run a controlled experiment, we studied how people use the system in five public workshops with 40 participants, aged 8 to 52, most of them new to head-mounted displays. In each workshop, eight participants formed two teams of four; while one team took part in an activity, the other watched the four views on a large screen with the players’ eye movements overlaid. After a short introduction, in which participants found their own view among the four and tried to shake hands with each other, they took part in two activities, reviewing the recordings with the researchers after each one.

Two screens of four first person views with colored eye movement traces, one from the game of tag and one from the drawing activity — The four views with each participant’s eye movements, as shown to the audience during the game of tag (left) and the drawing activity (right).

In drawing with faint memory, four participants sat at a desk and drew something almost everyone has seen but few can draw with confidence, such as a giraffe, the Statue of Liberty or a person portrayed on a banknote. They had no hints but the others’ views: they could hear each other but could not see the others’ drawings directly. In the game of tag, four participants walked among cardboard columns on a stage, and the chaser tagged the others with a soft sponge sword; the chaser’s view was highlighted in everyone’s goggles.

Drawing from each other’s memory

Four participants wearing cardboard goggles draw side by side at a long table — Drawing with faint memory, with Parallel Eyes.

The drawing activity overloaded participants with information, yet most of them developed ways of watching the others and completed their drawings. Each participant tended to pick a viewing target, often someone drawing the same or the next part, and changed it as the drawings changed: when the person P2 had been following erased part of their drawing, P2 switched to another participant and began drawing the dots found in that drawing. Participants alternated between a single focus on their target and a passing seek over all the drawings.

Four colored timelines, one per participant, showing whose view each participant was looking at over about 110 seconds of drawing a giraffe, with labels for when each part of the drawing appeared — Who each of four participants watched while drawing a giraffe. Each color shows whose view a participant was looking at; labels mark when each part of the drawing appeared, and the marked moments include the exchange of the head and horns (*1) and a change of viewing target (*4).

Through this interwoven reference, elements spread from drawing to drawing. One participant drew a head that another copied; the second added horns, and the first then added horns too. Two participants who did not recognize the person portrayed on a banknote still completed their drawings from what they saw in the others’ views.

Four hand-drawn giraffes from the same session, the first labeled with its parts: horn, face, outline of body, dot, tail, fore leg, back leg and hoof — The four giraffes from that session. Almost every element was drawn by each participant.

Animation of the four views during a drawing session

The paper reads this as a form of distributed cognition, in which people use each other to jog their memory, formed here without any verbal communication. Because no one knows whether the others are looking at their view, the setting removes social barriers that make people hesitate to refer to one another; the same property may also produce unexpected social relationships, which designers need to consider.

Seeing oneself from the chaser’s eyes

In the game of tag, people recognized their own body in the others’ views within a few minutes and went on to act on those views: they escaped by watching themselves from the chaser’s perspective, hid their own view while watching the others’, and decided where to go from another player’s view.

At first, many participants found it confusing even to walk. Once guided to find their own perspective, for instance by looking down at their own feet, they recognized their body and could even walk a bit faster. Some chased participants were startled to find themselves in the chaser’s view. As the games went on, they developed strategies and described them afterwards:

“What I did was I used the chaser’s perspective when I escaped from the chaser. Because I could see myself from the chaser’s point of view.” (P12)
“When I found myself in the chaser’s perspective, I knew it was time to escape!” (P4)
“I could use multiple views… When I got closer to the chaser, I could see the situation from another player’s view. So I decided where to escape to.” (P3)
“My particular tip is to hide your vision. I kept facing the floor, but I was watching the others’ views.” (P6)

Six moments of a game of tag, each showing four first person views with colored eye movement traces and marked events — Moments in the game of tag, with each participant’s eye movements traced in color. In (c), P4 escapes using the view of the chaser, P3 (*5); in (a), P3 hides their own view by facing the floor (*2).

Participants wearing the goggles walk among white panels on a stage during the game of tag; one holds a soft sword — The game of tag on a stage with cardboard columns.

Animation of a game of tag with paralleled shared views

Attention cues and context building

Looking again at these moments in the recordings and eye traces, we found that the strategies rested on two successive steps. The first is an attention cue: without having looked at a view, participants’ eyes jumped to something related to their own body, such as a small image of themselves, a moving object with the chaser’s color or an item with the color of their own clothes. The eye traces show these sudden movements, yet no participant mentioned them in the reflection; we hypothesize that peripheral vision plays a significant role in this unconscious perception. The second is context building, in which participants confirmed whether the cue was themselves, worked out from where the others were looking at them, and understood where the others were. Participants described this step explicitly: “As a chaser, I sometimes found somebody watching me, and I started to chase him.”

Keeping one’s own view among the shared ones appears to make this quick adaptation possible. Art projects that swap perspectives between people disconnect perspective from the body, and people trying them had difficulty walking or even moving their heads. Participants in Parallel Eyes always had their own view as one of the four; they recognized their bodies within a few minutes and built their viewing strategies on top of it.

Losing and restoring one’s own body

Following another person’s view could also cost people their own body. Some participants forgot to move and stood still while watching the others, and several walked into the cardboard walls while focusing on the chaser’s view. When we suddenly shuffled the layout of the four views before a game, participants lost track of their bodies, and those doing similar actions kept taking another person’s view for their own. To find themselves again, they moved their heads or waved their hands in front of their faces; a distinctive hand gesture, such as rock, paper or scissors held in front of the camera, worked well. The paper therefore suggests keeping one’s own view in a stable position, marking which view is whose when the layout has to change, and providing hand gestures that help users keep their own embodiment.

Beyond the shooter and the watcher

Video-mediated communication has mostly taken two forms: face-to-face video conferencing, modeled on people sitting or standing together, and first person video sharing, in which a remote person looks over a local person’s shoulder in fixed roles. Parallel Eyes forms a third configuration, a mutual first person video exchange, which is symmetric and goes beyond any physical metaphor. In this configuration, people do not necessarily need explicit verbal communication to understand each other, and the paper suggests it for uses where several people must make sense of a lot of information in parallel and coordinate their activities, such as professional services and emergency or security operations.

A table comparing face to face, first person video sharing and mutual first person video exchange by configuration, metaphor and use cases — Three configurations of video-mediated communication: face to face (F2F), first person video sharing (FPV) and mutual first person video exchange (FPX).

References

Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto. 2016. Parallel Eyes: Exploring Human Capability and Behaviors with Paralleled First Person View Sharing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1561–1572. https://doi.org/10.1145/2858036.2858495

BibTeX

@inproceedings{10.1145/2858036.2858495,
  author    = {Kasahara, Shunichi and Ando, Mitsuhito and Suganuma, Kiyoshi and Rekimoto, Jun},
  title     = {Parallel Eyes: Exploring Human Capability and Behaviors with Paralleled First Person View Sharing},
  booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '16},
  year      = {2016},
  pages     = {1561--1572},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/2858036.2858495}
}

Shunichi Kasahara, Mitsuhito Ando, Kiyoshi Suganuma, and Jun Rekimoto. 2016. Parallel eyes: exploring human capability and behaviors with paralleled first person view sharing. In ACM SIGGRAPH 2016 VR Village (SIGGRAPH ’16). Association for Computing Machinery, New York, NY, USA, Article 16, 1–2. https://doi.org/10.1145/2929490.2929495

BibTeX

@inproceedings{10.1145/2929490.2929495,
  author    = {Kasahara, Shunichi and Ando, Mitsuhito and Suganuma, Kiyoshi and Rekimoto, Jun},
  title     = {Parallel eyes: exploring human capability and behaviors with paralleled first person view sharing},
  booktitle = {ACM SIGGRAPH 2016 VR Village},
  series    = {SIGGRAPH '16},
  year      = {2016},
  articleno = {16},
  pages     = {1--2},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/2929490.2929495}
}

News: Parallel Eyes Workshop: Yamaguchi’s Future Classes at YCAM
Related projects: JackIn Head, Malleable Embodiment, Parallel Ping-Pong, Parallel Adaptation, ShadowClones

Exhibition

Ginza Sony Park #004, 未来の銀座の運動会, Parallel Eyes, 2018.12.22 – 2019.1.27, Tokyo Japan.
Talk Session, 田中章浩・渡邊克巳・大崎智史・笠原俊一・西田宗千佳, トークセッション「ヴァーチャル世界でワタシはどうなる？」, 日本科学未来館 5階コ・スタジオ, 日本未来館. 2018/07/29, Tokyo Japan.
Exhibition, SXSW 2017, Sony The WOW Factory, Interactive Festival, 3/12 – 3.14, Austin, Texas
Exhibition, Nico Nico Chokaigi (ニコニコ超会議) 2016, Parallel Eyes, at Makuhari Messe, Japan
Exhibition, TOKYO GAME SHOW 2016, Parallel Eyes, September 15 to 18 at Makuhari Messe, Japan