Evaluate Policies

armnet-policy-eval is the generic Armnet policy evaluation command. It keeps one user-facing CLI while dispatching to policy-specific runtime backends. Each backend still builds its own Docker image, so LeRobot, OpenPI, GR00T, and future policy stacks can keep separate dependencies.

LeRobot Policies

For LeRobot-compatible policies, use the same flags as armnet-lerobot-eval; only the command name changes:

armnet-policy-eval \
  --policy.path=pravsels/smolvla_open_lamp_door \
  --armnet.embodiment=lerobot/bimanual_so101 \
  --armnet.task=open_lamp_door \
  --eval.n_episodes=1 \
  --eval.batch_size=1 \
  --armnet.episode_time_s=30 \
  --armnet.fps=20

If --policy.type is omitted, the command defaults to the LeRobot backend.

OpenPI Policies

Select OpenPI with --policy.type=openpi. The generic command maps --policy.path and --policy.config_name onto the existing OpenPI runtime backend:

armnet-policy-eval \
  --policy.type=openpi \
  --policy.path=volume://openpi/checkpoints/5000 \
  --policy.config_name=pi05_so101_stacking_rings \
  --policy.actions_to_execute=25 \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task="stack the rings" \
  --armnet.n_episodes=3 \
  --armnet.episode_time_s=30 \
  --armnet.secrets="{HF_TOKEN: huggingface-token}"

OpenPI checkpoints are Orbax step folders containing params/ and assets/. Use a volume:// path, or upload a local checkpoint with the backend-specific volume upload flags.

GR00T Policies

Select NVIDIA GR00T N1.7 with --policy.type=groot:

armnet-policy-eval \
  --policy.type=groot \
  --policy.path=https://huggingface.co/pravsels/groot1.7_insert_candle \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task=insert_candle \
  --armnet.n_episodes=1 \
  --armnet.episode_time_s=30 \
  --armnet.secrets="{HF_TOKEN: huggingface-token}"

The GR00T backend accepts root-level Hugging Face model repos and nested checkpoint paths, for example:

pravsels/groot1.7_insert_candle
pravsels/groot1.7_insert_candle/checkpoints/8000/pretrained_model
https://huggingface.co/pravsels/groot1.7_insert_candle/tree/main/checkpoints/8000/pretrained_model

For bimanual SO101 checkpoints, route to a bimanual cell:

armnet-policy-eval \
  --policy.type=groot \
  --policy.path=pravsels/groot1.7_insert_candle \
  --armnet.embodiment=lerobot/bimanual_so101 \
  --armnet.task=insert_candle

Recorded Eval Datasets

Each backend records eval episodes as a LeRobot dataset. By default, recorded datasets are pushed to Hugging Face under:

<hf_user>/eval_<task>_<timestamp>

Pass HF_TOKEN through --armnet.secrets to push private datasets or to resolve the target Hub username. Disable uploads with --armnet.push_to_hub=False.