Skip to main content

Teleop

Teleop is two processes: a publisher that turns a live source into a robot reference, and the runtime, which subscribes to it over ZMQ. Robot state and commands never leave the runtime process.

Pico / XR  ──▶  pico_pub.py (retarget to the robot)  ──ZMQ :28701──▶  boltrt --task tracking

The publishers live in src/tasks/track/teleop/, a self-contained Python project with its own environment. They read robot names, joints, bodies and models from the same configs/robots/*.yaml as the runtime.

PublisherSourceUse
pico_pub.pyPico / XR whole-body tracking, retargeted with GMRlive teleoperation
npz_pub.pya retargeted NPZ cliptesting the teleop path without a headset

Set up the publisher​

uv sync --project src/tasks/track/teleop

For Pico, also install the XRoboToolkit PC Service and build its Python binding:

  1. Install the PC Service .deb from XRoboToolkit-PC-Service releases (sudo apt install ./XRoboToolkit_PC_Service_*.deb) and start XRoboToolkit from the app launcher.

  2. Build the SDK binding into the teleop environment. The script clones the SDK repositories at pinned commits into src/tasks/track/teleop/third_party/:

    src/tasks/track/teleop/setup_xrobot_sdk.sh
  3. On the headset: wear the trackers, finish whole-body calibration, connect XRoboToolkit to this machine and enable whole-body streaming.

Run​

Start the publisher, then the runtime:

# Terminal 1: Pico publisher (prints an mjviser URL showing the retargeted robot)
uv run --project src/tasks/track/teleop src/tasks/track/teleop/pico_pub.py --robot g1

# Terminal 2: runtime
boltrt --task tracking --robot g1 --policy policy.yaml --backend mujoco \
--motion tcp://127.0.0.1:28701

Without a headset, use a clip instead:

uv run --project src/tasks/track/teleop src/tasks/track/teleop/npz_pub.py \
--robot g1 --motion-path motions/walk1_subject1.npz --loop

If the publisher runs on another machine, point --motion at it: --motion tcp://<publisher-ip>:28701. Use --backend real for the real robot.

Controls​

ControlWhereEffect
XPico controllerPause / resume the published motion (handled by the publisher)
I, ]runtime terminalInit pose, enter policy
Qruntime terminalQuit

The publisher also sends Pico button states on port 5592 and analog trigger grips on port 5593 for other consumers; Bolt Runtime does not read them, so mode changes are made from the runtime terminal.

Ports​

PortContent
28701robot reference JSON (pico_pub.py, npz_pub.py)
28702SONIC SMPL reference (pico_pub.py --publish-smpl)
5592Pico button states (binary)
5593hand-grip triggers (binary)

Supported robots for Pico retargeting: g1, rp1 and rp1-24dof. Retargeting uses GMR onto the robot model from configs/robots/. G1 uses GMR's own IK config. The RP1 IK configs in src/tasks/track/teleop/boltrt_teleop/configs/ are adapted to the current RP1 models and have not yet been validated with a headset.

Robot reference messages​

JSON messages carry a robot-space reference:

FieldRequiredMeaning
joint_names, joint_posyesJoint positions by name
body_names, body_pos_w, body_quat_wyesBody poses in world frame, quaternions wxyz
joint_velnoJoint velocities
seq, publish_t_ns, pico_recv_time_nsnoSequencing and latency diagnostics
motion_first_framenoMarks the start of a new segment

Any publisher that sends these fields works with the runtime.

Behavior​

  • The runtime waits for the first valid message before tracking.
  • A new segment is aligned in XY position and yaw to stay continuous with the current reference.
  • If the stream stops, the last valid frame is held.
  • Invalid messages are logged and never overwrite a valid reference.

SMPL input​

SONIC SMPL policies consume a human reference instead of robot joints. Start the Pico publisher with --publish-smpl --smpl-human-joints-info-path <sonic-smpl-checkpoint>/human_joints_info.pkl and connect the runtime to port 28702. The source accepts SONIC's JSON or packed (1280-byte header) protocol with smpl_joint_pos_root (N×24×3), smpl_root_quat_w (wxyz) and robot joint_pos. The runtime does not solve or retarget human pose itself.