Shall We Dance?

Shall We Dance? Resonance of Intentions with an Embodied Agent based on the Free Energy Principle

Takeru Hashimoto, Jun Tani, and Shunichi Kasahara
ACM SIGGRAPH 2026 Emerging Technologies

The collaboration project with the leading author, Takeru Hashimoto and Jun Tani (Okinawa Institute of Science and Technology)

Paper (ACM Digital Library) · PDF

Abstract

Physical joint action shapes a sense of partnership by enabling individuals to negotiate intentions through mutual coordination. However, conventional physical agents typically rely on pre-learned transition rules, which limits their ability to adapt to a partner’s improvisational interventions. To address this, we propose a physical agent grounded in the Free Energy Principle that negotiates intentions through physical interaction. The agent uses a predictive coding-based RNN to dynamically update their internal intention (μ) and time constant (τ), minimizing real-time prediction errors caused by physical contact. This mechanism allows the agent to synchronize with the human’s motion pattern and tempo. In the demonstration “Shall We Dance?”, visitors hold hands with the agent and haptically perceive the resistance at the hands disappear as the agent infers the human’s new intent.

Negotiating intentions hand in hand

When two people move together, each keeps an intention of their own while inferring the other’s, and the joint movement is negotiated through contact. Most physical agents instead treat a mismatch with their partner as a control error to be corrected by the motors. Agents based on predictive coding have imitated turn-taking and mutual yielding, but they switch behavior according to transitions learned from training data, so they struggle when a partner improvises at an arbitrary moment. We asked whether an agent could follow a partner’s change of intention from any state, holding hands in a simple dance.

Three photos of a person holding a robot’s hands while dancing, above diagrams of the agent’s latent space with two attractors and a plot of prediction error and belief update over the phases T1 to T4 — A hand-in-hand dance in four phases: the human and the agent move in sync on one pattern (T1); the human changes intention and the agent’s prediction error rises (T2); the agent updates its belief, moving its internal state to another attractor (T3); both move in sync on the new pattern (T4).

How the agent works

The agent’s movement comes from a recurrent neural network trained so that each motion pattern becomes a stable loop, an attractor, in its internal state space. The network keeps predicting the next position of the hands. During the dance its weights stay fixed. When the partner’s actual movement differs from the prediction, the agent updates its internal state, its belief about the shared intention, so as to reduce the prediction error, following the Free Energy Principle. A large enough mismatch moves the state from one attractor to another, and the agent changes pattern. Updating the time constant in the same way lets the agent speed up or slow down to the partner’s tempo.

The prediction is also the target position for a custom two-degree-of-freedom force-feedback robot arm, so one loop serves both perception and action. Inference runs at 30 Hz on a laptop CPU, and the motor commands are filtered and sent to the actuators at 100 Hz.

Diagram: in training, an RNN learns two motion patterns as two attractors decoded into right and left hand trajectories; in inference, the error between prediction and observation from the real robot drives a belief update and the motor command — Training: the network learns two motion patterns as separate attractors. Inference: the error between predicted and observed movement drives both the belief update and the motor command.

Switching without learned rules

The agent has no rule for when or how to switch patterns: it changes its movement because the partner’s push makes its prediction wrong, so it can follow a change at any moment. In the validation, a person alternated the agent between two patterns, in phase and in antiphase, by applying force to its hands. Each push first raised the prediction error; the agent then updated its internal state, and both errors fell as it settled into the new pattern and moved in sync again. When the partner changed the frequency instead, the inferred speed ratio followed it while the pattern itself was kept.

Time-series plots of motion pattern labels, right and left hand position (observation and prediction), prediction error and belief update over 40 seconds, with blue bands where the human applied force — A partner switched the motion pattern by applying force (blue bands). Each time, the prediction error rose, the belief update followed, and both settled once the agent was in sync on the new pattern.

In the demonstration, visitors hold the robot’s hands with both hands. The agent starts one of its learned patterns and the visitor follows; once they are in sync, the visitor changes the pattern. The agent’s inference is felt through the palms: the hands stiffen when the intentions diverge, then suddenly go light as the agent takes up the visitor’s intention.

In the current system the agent’s impedance is fixed, so it does not ease its force while the two are negotiating, and the resistance lasts throughout the transition. Adjusting the agent’s cognitive and mechanical impedance to minimize free energy could let touch serve as a channel through which people infer the robot’s internal state more directly.

Reference

Takeru Hashimoto, Jun Tani, and Shunichi Kasahara. 2026. Shall We Dance? Resonance of Intentions with an Embodied Agent based on the Free Energy Principle. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies (SIGGRAPH Emerging Technologies ’26). Association for Computing Machinery, New York, NY, USA, 1–3. https://doi.org/10.1145/3799826.3812424

BibTeX

@inproceedings{10.1145/3799826.3812424,
  author    = {Hashimoto, Takeru and Tani, Jun and Kasahara, Shunichi},
  title     = {Shall We Dance? Resonance of Intentions with an Embodied Agent based on the Free Energy Principle},
  booktitle = {Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies},
  series    = {SIGGRAPH Emerging Technologies '26},
  year      = {2026},
  pages     = {1--3},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  doi       = {10.1145/3799826.3812424}
}

Figures from the paper (Hashimoto et al., SIGGRAPH ’26 Emerging Technologies), licensed under CC BY 4.0.

News: Shall We Dance? at SIGGRAPH 2026 Emerging Technologies
See also: https://tkrtkr.com/project/shall_we_dance/