Train and Deploy a LeRobot Policy

armnet-lerobot-eval is the Armnet-native equivalent of lerobot-eval for managed real-robot cells. It supports all default LeRobot policy architectures, as well as 3rd party plugins via the standard register_third_party_plugins() import system.

Evaluate a Hugging Face Policy

If your policy is public on the Hub, pass the repo ID directly:

armnet-lerobot-eval \
  --policy.path=lerobot/diffusion_pusht \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task=assemble_block_tower \
  --eval.n_episodes=3

Upload a Local Policy Checkpoint

If your policy checkpoint is not hosted on Hugging Face Hub, upload it to your Armnet Volume:

armnet volume upload \
  ./outputs/train/my_policy/checkpoints/005000/pretrained_model \
  policies/my_policy

Then refer to it with a volume:// path:

armnet-lerobot-eval \
  --policy.path=volume://policies/my_policy \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task=assemble_block_tower \
  --eval.n_episodes=10 \
  --eval.batch_size=1 \
  --armnet.episode_time_s=30

The runtime resolves volume://policies/my_policy to ctx.volume.path("policies/my_policy").

Evaluating Private Hugging Face Models

If your policy or dataset requires Hugging Face authentication, store your token as a Armnet Secret:

armnet secret create huggingface-token hf_...

Then request it in your job:

armnet-lerobot-eval \
  --policy.path=your-org/private-policy \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task=assemble_block_tower \
  --eval.n_episodes=10 \
  --eval.batch_size=1 \
  --secrets HF_TOKEN=huggingface-token

The runtime container receives HF_TOKEN as an environment variable, which Hugging Face libraries can use automatically.

Common Flags

armnet-lerobot-eval \
  --policy.path=volume://policies/my_policy \
  --policy.device=cuda \
  --policy.use_amp \
  --eval.n_episodes=10 \
  --eval.batch_size=1 \
  --eval.fps=20 \
  --eval.start_seed=0 \
  --armnet.episode_time_s=30 \
  --armnet.embodiment=lerobot/so-101 \
  --armnet.task=assemble_block_tower \
  --detach

Result Shape

The job returns:

{
  "policy_path": "...",
  "per_episode": [
    {"episode_ix": 0, "success": True, "duration_s": 12.3, "ticks": 246}
  ],
  "aggregated": {
    "pc_success": 100.0,
    "n_episodes": 1,
    "eval_s": 12.3
  }
}

Success is currently derived from the robot cell completion monitor via ctx.cell.is_complete().

Train with Background Armnet Evals

armnet-lerobot-train wraps LeRobot training and watches for new saved checkpoints. When a checkpoint appears, it uploads that checkpoint to your Armnet volume and submits a detached real-robot eval job in the background. Training continues even if the remote eval fails.

armnet-lerobot-train \
  --dataset.repo_id=your-org/your-dataset \
  --policy.type=act \
  --policy.repo_id=villekuosmanen/so101_red_cup \
  --output_dir=outputs/train/my_policy \
  --steps=20000 \
  --save_freq=2000 \
  --armnet.eval.embodiment=lerobot/so-101 \
  --armnet.eval.task=assemble_block_tower \
  --armnet.eval.n_episodes=10 \
  --armnet.eval.episode_time_s=30 \
  --armnet.eval.secrets HF_TOKEN=huggingface-token

Important notes:

• --output_dir is required so the wrapper can discover checkpoints.
• Remote evals are submitted with detach=True and stream_logs=False.
• The wrapper uploads checkpoints under a volume prefix such as policies/training-evals/<run_name>/<step_id>.
• If the wrapper process has an active wandb run, completed remote eval metrics are logged as remote_eval/*.

Hugging Face Checkpoint Transport

By default, armnet-lerobot-train uploads eval checkpoints to your Armnet volume. You can instead upload eval checkpoints to the policy's Hugging Face model repo:

armnet-lerobot-train \
  ... \
  --armnet.eval.use_hf_checkpoints

When enabled, each eval checkpoint is uploaded to a Hub branch named:

eval-step_005000

and the remote eval job is launched with:

policy_path=<policy repo id>
policy_revision=eval-step_005000

This keeps main available for the latest/final model while allowing old eval checkpoints to be reloaded by branch name.

Hugging Face as Checkpoint Storage

Using Hugging Face Hub for intermediate checkpoint storage is possible and may eventually be preferable to using Armnet volumes for policy checkpoints. LeRobot already pushes final policies through policy.push_to_hub. For intermediate checkpoints, the training wrapper would need to upload each pretrained_model checkpoint directory to the same model repo, for example under:

checkpoints/step_002000/pretrained_model/

Then armnet-lerobot-eval could evaluate that Hub path or revision. This is not implemented yet; the current MVP uses Armnet volumes for checkpoint transport.