Skip to content

Visualize a Job with Rerun

armnet can stream a running job's observations and actions to a live Rerun viewer on your machine — camera images, joint positions, commanded actions, and any scalars you choose. This is useful for watching real-robot evaluations as they happen.

How it works

container          ctx.log_rerun_data(observation, action)
                     -> protobuf packet, base64 on stdout behind a marker
robot cell         lifts the packet off stdout, publishes the raw protobuf
                     bytes on NATS  logs.<job_id>.rerun
orchestrator       WS /jobs/{job_id}/rerun forwards each packet as a binary frame
your machine       execute(use_rerun=True) decodes packets and replays them via
                     rr.log(...) into the viewer you started with rr.init(...)

The customer container never talks to NATS or Rerun directly. It only emits packets; the cell relays them and the client (your orchestrate script) is where the viewer runs and rr.log is called. Images are JPEG-compressed before they hit the message bus to keep the stream light.

Install the viewer

The Rerun SDK is an optional client dependency:

uv pip install 'armnet-client[viz]'   # or: pip install rerun-sdk

Nothing extra is needed in your container image beyond what already provides numpy/opencv (e.g. a LeRobot install) for image compression.

Runtime side: log data

Inside your @main runtime function, call ctx.log_rerun_data(...) with the observation and/or action you want to visualize. Keys are namespaced with observation. / action. automatically.

from armnet_runtime import Context, main


@main
def run(ctx: Context) -> dict:
    robot = connect_robot(ctx)
    for _ in range(num_steps):
        obs = robot.get_observation()
        action = policy.select_action(obs)
        robot.send_action(action)
        if ctx.args.get("use_rerun"):
            ctx.log_rerun_data(observation=obs, action=action)
    return {"ok": True}
  • Scalars become Rerun scalars.
  • Image-like arrays (CHW or HWC with 1/3/4 channels) become images.
  • 1-D / higher-D arrays become one scalar per element ({key}_{i}).
  • compress_images=True (default) JPEG-encodes images; pass compress_images=False to send raw RGB.

Gate the logging on a job arg (e.g. use_rerun) so the same runtime image works with and without visualization.

Client side: start the viewer and stream

Call init_rerun(...) (a thin wrapper over rr.init(..., spawn=True)) before submitting, then pass use_rerun=True to execute(...):

from armnet_client import execute, init_rerun

init_rerun("my-eval", spawn=True)

result = execute(
    image=image,
    embodiment="lerobot/so-101",
    task="assemble_block_tower",
    args={"use_rerun": True},   # consumed by your runtime function
    use_rerun=True,             # tells the client to stream into the viewer
    timeout_seconds=1200,
)

use_rerun=True starts a background thread that consumes the rerun websocket and replays packets into the viewer. It runs alongside the normal log stream and never blocks or fails the job — if the SDK is missing or the socket can't be reached, streaming is silently skipped.

Headless machines (SSH / no display)

init_rerun(..., spawn=True) opens a native desktop window, which needs a display. On a headless host you'll otherwise see:

Error: winit EventLoopError: ... neither WAYLAND_DISPLAY nor WAYLAND_SOCKET nor DISPLAY is set.

When no display is detected, init_rerun automatically serves Rerun's web viewer (serve_grpc() for the data + serve_web_viewer() for the UI) and prints how to reach it:

[armnet] Rerun viewer ready:
  web viewer:  http://localhost:9090/?url=rerun+http://localhost:9876/proxy
  or native:   rerun --connect rerun+http://localhost:9876/proxy
  on a remote host, forward BOTH ports first ...
      ssh -L 9090:localhost:9090 -L 9876:localhost:9876 <user>@<host>

Forward both ports

The viewer UI loads from the web port (9090), but it streams data from a separate gRPC port (9876). If you forward only 9090, the page loads but stays empty because the browser can't reach the data server. Open the full http://localhost:9090/?url=... URL (the bare URL won't auto-connect), or skip the browser entirely and connect a locally-installed native viewer with rerun --connect rerun+http://localhost:9876/proxy.

You can also force the web viewer explicitly on any machine:

init_rerun("my-eval", serve_web=True, web_port=9090, grpc_port=9876)

Full example

examples/orchestrate_so101_rerun.py submits the SO-101 smoke job with visualization enabled (reusing the existing runtime_so101_smoke image, which logs to Rerun only when args["use_rerun"] is set):

uv pip install 'armnet-client[viz]'
uv run python examples/orchestrate_so101_rerun.py

A Rerun window opens and shows the camera feed, measured joint positions, and commanded actions as the arm moves.

Notes

  • The NATS subject is logs.<job_id>.rerun; packets are serialized armnet.rerun.RerunPacket protobufs (see armnet_runtime.rerun).
  • Packets are dropped if no client is subscribed, so leaving use_rerun on in the runtime is cheap when nobody is watching.
  • ctx.log_rerun_data(...) is non-blocking: it hands a snapshot to a background thread that does the encoding and IO, so your control loop never stalls on visualization. The thread's queue is bounded and drops the oldest pending frame under backpressure — a slow consumer sheds frames instead of slowing the robot loop. Tune the buffer with ARMNET_RERUN_QUEUE_MAXSIZE (default 30).
  • High image resolution / frequency increases bandwidth; lower jpeg_quality (e.g. ctx.log_rerun_data(..., jpeg_quality=50)) or log fewer frames if the stream can't keep up.