How to run a clean benchmark with network limits against rubric leaks with harness2

Authorization notice. harness2 is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forResearcher reporting harness-improvement numbers who must show the agent could not read public rubrics

The problemExecutors have open web access and public rubrics may exist online, so scores can be inflated by looking up answers.

The outcomeA benchmark run done under network restrictions, giving numbers you can defend as not contaminated by leaked grading criteria.

What you need first

• Linux, Python 3.12 and uv for installation
• No credentials needed for the local demo (harness2-demo)
• Model-backed runs need Node.js 20+, the Google Cloud CLI, a Vertex AI project and model IDs in .env, and gcloud application-default login
• LAB needs Pandoc and Poppler; WorkBuddy-Bench needs Docker
• Model-backed benchmark runs incur API costs

Steps

1. Read the web access section of docs/EXPERIMENTS.md and apply its network restrictions.
2. Preview task selection with no model calls by adding --dry-run to your harness2 command.
3. Run the real refinement: harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1
4. Judge: harness2-judge --bench lab --k 1 --mode parallel
5. Report: harness2-report --bench lab --k 1 --mode parallel

Prompt for your agent

Before running any model-backed benchmark in google-research/harness2, read the web access section in docs/EXPERIMENTS.md and tell me what network restrictions to configure for <benchmark>. Then run a --dry-run preview and verify task selection looks right.

Caveat. Authorized use only: run this only inside infrastructure you control, since the README states executors have unrestricted network access and Codex runs with its sandbox disabled, and public rubrics may be online.

• Markdown version
• JSON version