DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to re-run one scenario family to debug a harness with Doc2 Harness

Narrow scoring to one family, such as headless or permission, to find a failing area.

Who it is forA developer whose full catalog run flagged failures and who wants to iterate on one area of the harness quickly.

The problemRe-running all 94 scenarios after every fix wastes time when the failures sit in a single area like permissions or the terminal interface.

The outcomeA focused score and per-scenario receipts for one family, so fixes can be checked against only the relevant scenarios.

What you need first

  • Python 3.11 or newer
  • Docker installed and running
  • The repo cloned locally
  • For building a new harness only: the builder image, a local LiteLLM gateway, and your own API key

Steps

  1. Complete the evaluation image build from the Quick start section of the README.
  2. Run PYTHONPATH=src:tools:run python3 tools/run_production_catalog.py --artifact /path/to/product --artifact-id my-product --output /tmp/scores --concurrency 12 --timeout 300 --family headless and replace headless with the family you want (for example permission). (inferred, check the README)
  3. Open the receipt JSON files in /tmp/scores for the scenarios that failed.
  4. Fix the harness and run the same command again to confirm the change.

Prompt for your agent

Use the repo YuyaoGe/doc2harness to re-run only the <family-name> scenario family against the product at <path-to-product-directory>. Use tools/run_production_catalog.py with --family <family-name> and report which scenarios failed. Verify the catalog still loads with the family filter applied.

Caveat. The README says to keep a narrowed selection closed under each scored scenario's declared controls, so a filtered run may need extra scenarios added back.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page