DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to benchmark a vision-language-action policy in simulation with Lerobot

Evaluate a pretrained Pi0 policy on the LIBERO benchmark with lerobot-eval.

Short answer: Install with pip install lerobot; then run lerobot-eval with --policy.path=lerobot/pi0_libero_finetuned, --env.type=libero, --env.task=libero_object and --eval.n_episodes=10; Swap in another policy path or benchmark such as MetaWorld to compare. All steps.

Who it is forA researcher comparing VLA models before trying one on a real arm.

The problemTesting policies on real robots is slow and risky without a simulation baseline.

The outcomeSuccess results over 10 LIBERO episodes for a published policy.

What you need first

  • Python and pip
  • Extras for your task: lerobot[dataset], lerobot[training], or lerobot[core_scripts,feetech] for an SO-101 arm
  • A GPU for training most policies (see the Compute Hardware Guide)
  • Optional: a supported robot such as SO101, Koch or LeKiwi, and a Hugging Face account to push datasets and models

Steps

  1. Install with pip install lerobot
  2. Run lerobot-eval with --policy.path=lerobot/pi0_libero_finetuned, --env.type=libero, --env.task=libero_object and --eval.n_episodes=10
  3. Swap in another policy path or benchmark such as MetaWorld to compare
  4. Use lerobot-rollout, per the Policy Deployment Documentation, once you want to run a trained policy on a real robot
  5. Publish your own environment through EnvHub if you build a new benchmark

Prompt for your agent

With huggingface/lerobot, run lerobot-eval on --policy.path=lerobot/pi0_libero_finetuned with --env.type=libero --env.task=libero_object --eval.n_episodes=10, then summarise the success rate.

Caveat. Simulation success does not guarantee the same behaviour on a physical robot.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for huggingface/lerobot and "Benchmark a vision-language-action policy in simulation"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer