How to compare parallel and sequential refinement on the same task with harness2

Authorization notice. harness2 is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forML engineer deciding how to spend an agent-refinement compute budget on a recurring task type

The problemIt is unclear whether exploring many candidates at once or refining step by step gives better results per dollar for your task.

The outcomeTwo reports for the same task, one per mode, that you can compare side by side to choose a strategy.

What you need first

• Linux, Python 3.12 and uv for installation
• No credentials needed for the local demo (harness2-demo)
• Model-backed runs need Node.js 20+, the Google Cloud CLI, a Vertex AI project and model IDs in .env, and gcloud application-default login
• LAB needs Pandoc and Poppler; WorkBuddy-Bench needs Docker
• Model-backed benchmark runs incur API costs

Steps

1. Run parallel mode: harness2 --bench lab --k 2 --mode parallel --only-domains antitrust-competition --max-tasks 1
2. Run sequential mode: harness2 --bench lab --k 2 --mode sequential --only-domains antitrust-competition --max-tasks 1
3. Judge each run: harness2-judge --bench lab --k 2 --mode parallel and harness2-judge --bench lab --k 2 --mode sequential
4. Report each run: harness2-report --bench lab --k 2 --mode parallel and harness2-report --bench lab --k 2 --mode sequential
5. Compare the two reports and the adopted harnesses.

Prompt for your agent

Using google-research/harness2, run LAB on one task with --k 2 in both parallel and sequential modes, judge and report each, and tabulate the scores side by side. Verify both runs used the same task selection and seed before comparing.

Caveat. A single task is a very noisy comparison, and the README ties each judge and report call to its own mode, k and seed, so run the full command set once per mode.

• Markdown version
• JSON version