DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to refine an agent harness on a small LAB legal set with Harness2

Run one Propose-Probe-Compose step on a single LAB legal domain, then judge and report the result.

Credentials. The setup uses gcloud application-default login and a .env file with Vertex AI settings, so keep those credentials out of shared logs and commits.

Who it is forA legal-tech or AI evaluation engineer who needs evidence on whether a harness refinement helps an agent on legal work before scaling up.

The problemJudging whether a harness change helps requires running real model-backed tasks, which is expensive and easy to misconfigure.

The outcomeA judged and reported LAB run for one domain with a small task count, showing whether the refined harness improved results.

What you need first

  • Linux, Python 3.12, and uv
  • For the local demo: nothing else, since it needs no model credentials
  • For model-backed runs: Node.js 20+, the Google Cloud CLI, a Vertex AI project with model IDs, and API budget
  • For LAB: Pandoc and Poppler. For WorkBuddy-Bench: Docker

Steps

  1. Create the environment with uv venv --python 3.12 .venv, activate it, and install with uv pip install -e '.[render]'
  2. Run cp .env.example .env, edit .env with your Vertex AI project and model IDs, then run gcloud auth application-default login
  3. Install the LAB benchmark with harness2-setup lab and check it with harness2-check --bench lab
  4. Run one task with harness2 --bench lab --k 1 --mode parallel --only-domains antitrust-competition --max-tasks 1; add --dry-run first to preview task selection without model calls (inferred, check the README)
  5. Judge the run with harness2-judge --bench lab --k 1 --mode parallel
  6. Generate the summary with harness2-report --bench lab --k 1 --mode parallel

Prompt for your agent

In the google-research/harness2 repo, set up the LAB benchmark, then run harness2 --bench lab --k 1 --mode parallel --only-domains <domain> --max-tasks 1, followed by harness2-judge and harness2-report with the same flags. Report the judged result for the task and the path of the saved results under runs/. Verify that the judge and report commands used the same benchmark, k, mode, and domain as the refinement run.

Caveat. Model-backed runs incur API costs, so start with the single task limit shown and keep the same flags across run, judge, and report.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page