Beyond Motion Tracking:Steerable Physical
Interaction Competence

Consider humanoid whole-body control, my main research area. (The discussion below also applies to dexterous hands.) Motion tracking in free space, without object interaction or complex terrain, is already very mature. Trackers such as SONIC and Mimic-Lite can naturally follow diverse whole-body motion references. Work such as Omega-0, Delta-0, and Archon connects these trackers to VLAs, allowing S1 to command coordinated whole-body actions such as moving, reaching, and bending.

Start with these three demos.

Now compare them with PHP, Workhorse, and Boston Dynamics’ Atlas below.

The policies in PHP and Workhorse feel highly responsive: they adjust their behavior in real time as the terrain changes or the objects they handle are perturbed. Atlas’s throw makes the object’s weight and motion visibly part of the whole-body action. Some whole-body VLA demos accomplish complex movements, yet show less of this responsiveness to changes in the interaction.

This impression draws my attention to two limitations of S0. First, in a common motion-centric architecture, S1 observes the environment and sends body-motion references through a motion interface. S0 receives no scene input and tracks those references primarily from proprioception. When the environment changes, that information must first pass through S1 and become a new motion reference before it can affect S0’s execution. S0 lacks its own loop for directly perceiving and responding to external interaction.

Second, consider S0’s training experience. An S0 trained mainly to track body motion in free space, without experiencing the relevant object dynamics, will not reliably throw a heavy object with its whole body just because it receives a beautiful throwing reference. The object’s mass, inertia, and contacts affect balance and motion. S0 needs training experience that teaches it to handle these effects. Neither faster reasoning in S1 nor more accurate motion references can replace that dynamics grounding.

PHP, Workhorse, and Boston Dynamics’ Atlas share one feature: S0 accounts for object and environment dynamics in training or control. The RL policies take scene- or object-related information as input, execute their own actions in simulation, and learn subsequent control from the resulting feedback. Boston’s MPC instead explicitly models object mass, momentum, and the forces applied by the robot, jointly considering how the robot and object will move. The former acquires dynamics grounding through closed-loop experience; the latter uses explicit physical modeling. Both make S0 account for how its actions affect the outside world, rather than simply reproduce body motion.

But these S0s are all task-specific controllers. PHP focuses on terrain locomotion and parkour; Workhorse’s website explicitly states that each task has its own planner and tracker. These are valuable capabilities, but they do not yet provide the general S0 I want: different tasks, objects, and contact modes handled by the same controller through a common interface.

The capability I want to emphasize is responsiveness: can S0 promptly adjust its behavior when contact, load, or the environment changes? On the sim-RL path I want to pursue, S0 must experience how its own actions affect objects and the environment, and learn to adjust from that feedback. A policy that has never experienced object-interaction dynamics cannot learn the corresponding closed-loop control from free-space tracking alone. Better tracking accuracy or better motion references from S1 cannot fill that missing interaction experience.

How do we acquire interaction competence at scale?

Data is certainly the most important ingredient. Beyond data, I am most interested in representations that let us scale dynamics grounding across diverse objects and environments. These have two parts: scene awareness, how S0 perceives the world it interacts with; and interaction intent, how we represent the desired physical behavior for S0 to learn and execute.

HDMI is my work from a year ago. I wanted robots to do more than imitate human body motion: they should actually move objects. At the time, we considered robot–object co-tracking for an arbitrary single object and a single motion. As I scale this up, I have been rethinking the two representations: scene awareness for perceiving the world, and interaction intent for communicating the desired interaction.

What should these representations provide? Scene awareness should capture external information as completely as possible, using information available on a real robot. Interaction intent should describe the desired object motion as completely as possible. Both need to scale without being tied to predefined object frames.

In HDMI, I represented object information as an object pose in the robot frame. This works for a single object, but requires us to define that object’s coordinate frame in advance. An object should not inherently need a coordinate frame attached to it.

Point clouds, RGB, and depth offer a scalable alternative. The input is no longer tied to a fixed set of object definitions; it can provide finer-grained information and let S0 perceive the entire environment through a common representation. Scene input lets S0 see the outside world; training with object dynamics teaches it how to interact with that world. Continuous environmental feedback lets S0 adjust to the interaction as it unfolds, closing the loop with the outside world.

Recently, I have also become interested in contact-centric interfaces. Object motion alone does not fully specify an interaction. Sometimes the goal is to apply force without moving the object. When wiping a blackboard, for example, we need contact and force without moving the board. If the interface cannot specify contact, the end effector might merely brush past the surface without pressing hard enough to erase anything.

Body motion alone is also insufficient. Consider sitting on a chair: the same body motion could mean maintaining a seated posture through leg effort, or relaxing and letting the chair bear the load. If the interface can specify pelvis–chair contact, S0 can deliberately transfer weight to the chair instead of continuing to support it through the legs. The body motion is the same; the load-bearing contact and physical outcome are different. Without this distinction, a motion reference is ambiguous: it cannot tell the policy whether to keep supporting itself through the legs or rely on the chair.

Making contact requirements explicit gives S0 more complete information about the interaction it should execute: whether to maintain contact, the desired force or wrench, a force range, or compliance. Combined with robot motion, these targets distinguish interactions that would otherwise share the same motion reference.

I do not think object motion and contact intent must be alternatives. Object motion specifies the desired outcome, contact intent specifies the interaction, and robot motion describes how the body participates. Making contact intent work in practice still requires force sensing or estimation, suitable hardware, and solutions to the sim-to-real gap.

As LLMs and VLMs develop stronger spatial understanding, I expect S1 to specify finer-grained interaction intent. S0 should be able to execute those distinctions and produce the corresponding physical effects: whether to make contact, when to comply or release, and where an object should move next.

π0.7 and Steerable Policies both emphasize richer conditioning as a way to steer VLAs in more detail. I think the same idea applies to S0: S0 also needs to expose a more expressive interface.

This does more than make existing capabilities easier for S1 to use. For S0’s own learning, more complete conditioning reduces ambiguity: it can learn how to execute different intents instead of guessing how to interact from an incomplete interface. For S1, a richer interface allows more diverse and finer-grained interaction intent to reach S0. Steerability helps S0 learn the distinctions and S1 express them.

Dynamics grounding teaches S0 to execute interactions; steerability lets S1 direct those interactions in detail. The former comes from S0’s closed-loop experience across diverse objects, environments, and contact conditions, building responsiveness to interaction changes. The latter exposes those capabilities to S1. As spatial grounding improves in LLMs and VLMs, we need an S0 that can execute richer interaction intent and be steered through an expressive interface.

Imitating motion is an important starting point. Next, I want the same S0 to handle a broader range of real interactions: perceiving them, responding promptly, and realizing S1’s interaction intent in the physical world.