Skip to content

Evaluate an OpenPI Policy

armnet-openpi-eval is the OpenPI counterpart of armnet-lerobot-eval: it evaluates an OpenPI policy (pi0 / pi0.5) on a managed real-robot SO-101 cell instead of a LeRobot policy. Each episode is recorded as a LeRobot Dataset and (optionally) pushed to the Hugging Face Hub so you can inspect the rollout afterwards.

Prerequisites

  • An OpenPI checkpoint — an Orbax step folder containing params/ and assets/.
  • That checkpoint already on your Armnet Volume (upload it out-of-band, or let the CLI upload a local copy for you — see below).
  • The OpenPI training config name registered in the installed openpi (for example pi05_so101_stacking_rings).
  • A Hugging Face token stored as a Armnet Secret if you want the recorded eval dataset pushed to the Hub.

Evaluate a Checkpoint Already on Your Volume

Point --openpi.checkpoint_dir at the Orbax step folder on your volume with a volume:// path:

armnet-openpi-eval \
  --openpi.checkpoint_dir=volume://openpi/checkpoints/5000 \
  --openpi.config_name=pi05_so101_stacking_rings \
  --armnet.n_episodes=3 \
  --armnet.episode_time_s=30 \
  --armnet.fps=20 \
  --armnet.task="stack the rings" \
  --armnet.secrets="{HF_TOKEN: huggingface-token}"

The runtime resolves volume://openpi/checkpoints/5000 to ctx.volume.path("openpi/checkpoints/5000") inside the cell container.

Upload a Local Checkpoint While Evaluating

If the checkpoint only exists on your machine, upload it as part of the run with --armnet.volume_checkpoint_path. The CLI uploads the local --openpi.checkpoint_dir to that volume path first, then evaluates from there:

armnet-openpi-eval \
  --openpi.checkpoint_dir=./checkpoints/pi05_rings/5000 \
  --armnet.volume_checkpoint_path=openpi/checkpoints/5000 \
  --openpi.config_name=pi05_so101_stacking_rings \
  --armnet.n_episodes=3 \
  --armnet.task="stack the rings" \
  --armnet.secrets="{HF_TOKEN: huggingface-token}"

Pass --armnet.volume_overwrite=True to replace files that already exist on the volume.

Recorded Eval Datasets

Every episode is recorded into a LeRobotDataset. By default the dataset is pushed to the Hub under an auto-generated repo id:

<hf_user>/eval_<task>_<timestamp>
  • The HF_TOKEN you pass through --armnet.secrets is used both to push the dataset and to resolve <hf_user> from the token.
  • Override the namespace with --armnet.hf_user, or set an explicit repo id with --armnet.record_dataset_repo_id.
  • Disable the upload entirely with --armnet.push_to_hub=False.

Common Flags

armnet-openpi-eval \
  --openpi.checkpoint_dir=volume://openpi/checkpoints/5000 \
  --openpi.config_name=pi05_so101_stacking_rings \
  --openpi.actions_to_execute=25 \
  --openpi.device=cuda \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task="stack the rings" \
  --armnet.language_instruction="stack the rings on the peg" \
  --armnet.n_episodes=10 \
  --armnet.episode_time_s=30 \
  --armnet.fps=20 \
  --armnet.secrets="{HF_TOKEN: huggingface-token}" \
  --seed=1000 \
  --armnet.detach=True
Flag Default Purpose
--openpi.checkpoint_dir (required) volume:// path to the Orbax step folder (or a local path when uploading).
--openpi.config_name (required) OpenPI training config registered in the installed openpi.
--openpi.actions_to_execute 25 Actions consumed from each predicted action chunk before re-querying the policy.
--openpi.device cuda Inference device on the cell (OpenPI requires a CUDA-capable cell).
--armnet.embodiment lerobot/so-101 Target cell type: lerobot/so-101 (single arm) or lerobot/bimanual_so101 (dual arm). Must match the cell.
--armnet.task (none) Cell task slug; omit to let any cell of the embodiment pick the job up.
--armnet.n_episodes 1 Number of eval episodes.
--armnet.episode_time_s 30 Per-episode cap in seconds.
--armnet.fps 20 Policy/control rate on the cell.
--armnet.language_instruction (none) Instruction passed to the policy; overrides the cell's assigned-task instruction (set from the FMS/DB) for this job.
--armnet.detach False Return immediately instead of streaming logs and blocking until done.

CUDA-capable cells

OpenPI policy inference runs on the cell's GPU, so the job must land on a CUDA-enabled cell (docker_gpus set in the cell's robot_cell.json, or ARMNET_DOCKER_GPUS=all).

Bimanual SO-101 cells

Set --armnet.embodiment=lerobot/bimanual_so101 to evaluate on a dual-arm cell (e.g. cell 8). The runtime builds a BiSOFollower and feeds the policy a 12-DOF state (left arm joints then right arm joints) plus every camera image (top, left_wrist, right_wrist); your OpenPI config's repack transform selects the keys it was trained on. The job's embodiment must match the cell's arm count or the run fails fast with a clear error.

See Also