Embodied Agile Robotic Arm

AgiLimb: Embodied Agile Robotic Arm by Integrating Digital Reflex and Seamless Action Takeover

Yusuke Kawashima, Yoshiaki Abe, Takeru Hashimoto, Shunichi Kasahara, and Ryuma Niiyama
ACM SIGGRAPH Asia 2024 Emerging Technologies

The collaboration project with the leading lab of Ryuma Niiyama, Yusuke Kawashima and Yoshiaki Abe (Meiji Univ.) and Takeru Hashimoto (Sony CSL).

Paper (ACM Digital Library) · PDF

Video: AgiLimb catching and throwing a ball with digital reflex and enhanced action (1.7 min, with audio).

Abstract

Performing sports skills with a human-operated robot is a challenging task. The key to human-robot-human (HRH) interaction is to combine the physical capabilities of a robot with the flexibility of a human. We developed the AgiLimb system to enable operators to transcend their own perceptual and physical abilities in a robotic experience while maintaining a sense of agency. The system introduced fast vision-based digital reflex to overcome the delay in human higher-order visual feedback. In addition, the system incorporates action takeover mechanism to improve the accuracy of the motion. The basis for agile motion is a quasi-direct-drive robot manipulator designed specifically for this application. In the demonstration, visitors will experience operating the AgiLimb system that moves and reacts like their own body.

A fast arm held back by human reaction

With real-time motion capture, a robot arm can serve as an avatar that moves faster and more precisely than its operator. The AgiLimb arm is built for this: four quasi-direct-drive motors drive a 337 mm upper arm and a 284 mm forearm, which follow the operator’s motion-captured arm at 100 Hz and can accelerate and decelerate beyond human ability.

Following the operator, however, wastes most of that speed. About 250 ms pass between a visual stimulus and the onset of a person’s movement, so an arm that waits for its operator reacts no faster than the operator does. Its joints also differ from human ones, which makes a throw hard to aim as intended.

Three photos: the robot arm with a lacrosse head; the arm catching a ball thrown at it while the operator raises his hand; the arm throwing a ball that knocks over a bottle — Catch and throw with the human-operated robot arm. Left: the arm with a lacrosse head. Middle: catching a thrown ball. Right: throwing a ball at a bottle.

Handing over control for a moment

AgiLimb gives the arm to automatic control only for the moment a reflex or a precise throw is needed, and then hands it back to the operator. For catching, a digital reflex takes over: two cameras running at up to 140 fps track the red ball, triangulate its position and predict its trajectory with a kinematic model. The arm starts moving about 60 ms after tracking, faster than the human visual reflex. For throwing, an enhanced action takes over: the system detects the intention to throw from the operator’s arm, predicts the next 300 ms of the motion with a minimum-jerk model to estimate direction and speed, and generates a throw that suits the arm.

System overview: object tracking cameras and motion capture feed a computer; a local loop (digital reflex) predicts the ball trajectory and controls the arm; a teleoperation loop (enhanced action) estimates the operator’s intention — System overview. In the local loop, the digital reflex predicts the ball’s trajectory and moves the arm to catch it. In the teleoperation loop, the enhanced action estimates what the operator intends and generates the throw.

The handover is a blend rather than a switch. Each joint command mixes manual and automatic control with a ratio that stays at zero (full manual control) until a catch or a throw is detected. The ratio then rises at once, and after the motion it falls gradually back to zero, so control returns to the operator smoothly. Keeping the ratio below one leaves room for the operator to adjust the motion. The aim is to let operators act beyond their own perceptual and physical limits while keeping the sense that they are the ones acting.

Reference

Yusuke Kawashima, Yoshiaki Abe, Takeru Hashimoto, Shunichi Kasahara, and Ryuma Niiyama. 2024. AgiLimb: Embodied Agile Robotic Arm by Integrating Digital Reflex and Seamless Action Takeover. In SIGGRAPH Asia 2024 Emerging Technologies (SA ’24). Association for Computing Machinery, New York, NY, USA, Article 2, 1–2. https://doi.org/10.1145/3681755.3688951

BibTeX

@inproceedings{10.1145/3681755.3688951,
  author    = {Kawashima, Yusuke and Abe, Yoshiaki and Hashimoto, Takeru and Kasahara, Shunichi and Niiyama, Ryuma},
  title     = {AgiLimb: Embodied Agile Robotic Arm by Integrating Digital Reflex and Seamless Action Takeover},
  booktitle = {SIGGRAPH Asia 2024 Emerging Technologies},
  series    = {SA '24},
  year      = {2024},
  articleno = {2},
  pages     = {1--2},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3681755.3688951}
}

News: AgiLimb at SIGGRAPH Asia 2024 Emerging Technologies
Related projects: Parallel Ping-Pong