# AI use cases for Doc2 Harness (YuyaoGe/doc2harness) Repo: [YuyaoGe/doc2harness](https://github.com/YuyaoGe/doc2harness) | Use cases: 4 | Page: https://dedupedai.com/use-cases/yuyaoge-doc2harness/ Doc2Harness is a Python benchmark that measures how well a model builds a working coding agent harness, then tests that harness against a scripted mock model using 94 released scenarios. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Score an existing coding agent harness: https://dedupedai.com/use-cases/yuyaoge-doc2harness/score-existing-agent-harness/ - Re-run one scenario family to debug a harness: https://dedupedai.com/use-cases/yuyaoge-doc2harness/debug-one-scenario-family/ - Confirm the scenario catalog is intact before scoring: https://dedupedai.com/use-cases/yuyaoge-doc2harness/check-catalog-integrity-before-scoring/ - Build a new agent harness and score it: https://dedupedai.com/use-cases/yuyaoge-doc2harness/build-and-score-new-harness/ ## How to score an existing coding agent harness with Doc2 Harness Repo: [YuyaoGe/doc2harness](https://github.com/YuyaoGe/doc2harness) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Run all 94 released scenarios against your product directory and get a pass rate. **Who it is for:** A developer who has a coding agent product in a directory and wants an objective pass rate before a release or comparison. **The problem:** Checking whether a coding agent harness really enforces its rules and behaves correctly across file, permission, state, and startup cases is slow to do by hand. **The outcome:** A JSON receipt for each scenario plus catalog_receipt.json, with a score of pass divided by pass plus fail over applicable scenarios. ### Steps 1. Run `docker build --target evaluation -t doc2harness/evaluation:candidate -f containers/Dockerfile .` from the repo root (see containers/README.md for mirror options). 2. Confirm your product directory contains src/yuyao/, then run `PYTHONPATH=src:tools:run python3 tools/run_production_catalog.py --artifact /path/to/product --artifact-id my-product --output /tmp/scores --concurrency 12 --timeout 300`. (inferred, check the README) 3. Wait about 25 to 90 seconds at concurrency 12 for the full catalog to finish. 4. Open /tmp/scores/catalog_receipt.json and read the score, remembering that INCONCLUSIVE is not counted as a failure. ### Prompt to paste into your agent ```text Use the repo YuyaoGe/doc2harness to score the product at . Run tools/run_production_catalog.py with --artifact and --output , then report the pass and fail counts from catalog_receipt.json. Verify that the product directory contains src/yuyao/ before running. ``` ### Caveat The README says the leaderboard numbers come from the full 328-case suite and cannot be reproduced from this 94-scenario subset alone. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/yuyaoge-doc2harness/score-existing-agent-harness/ ## How to re-run one scenario family to debug a harness with Doc2 Harness Repo: [YuyaoGe/doc2harness](https://github.com/YuyaoGe/doc2harness) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Narrow scoring to one family, such as headless or permission, to find a failing area. **Who it is for:** A developer whose full catalog run flagged failures and who wants to iterate on one area of the harness quickly. **The problem:** Re-running all 94 scenarios after every fix wastes time when the failures sit in a single area like permissions or the terminal interface. **The outcome:** A focused score and per-scenario receipts for one family, so fixes can be checked against only the relevant scenarios. ### Steps 1. Complete the evaluation image build from the Quick start section of the README. 2. Run `PYTHONPATH=src:tools:run python3 tools/run_production_catalog.py --artifact /path/to/product --artifact-id my-product --output /tmp/scores --concurrency 12 --timeout 300 --family headless` and replace headless with the family you want (for example permission). (inferred, check the README) 3. Open the receipt JSON files in /tmp/scores for the scenarios that failed. 4. Fix the harness and run the same command again to confirm the change. ### Prompt to paste into your agent ```text Use the repo YuyaoGe/doc2harness to re-run only the scenario family against the product at . Use tools/run_production_catalog.py with --family and report which scenarios failed. Verify the catalog still loads with the family filter applied. ``` ### Caveat The README says to keep a narrowed selection closed under each scored scenario's declared controls, so a filtered run may need extra scenarios added back. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/yuyaoge-doc2harness/debug-one-scenario-family/ ## How to confirm the scenario catalog is intact before scoring with Doc2 Harness Repo: [YuyaoGe/doc2harness](https://github.com/YuyaoGe/doc2harness) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README Verify that the catalog loads and contains 94 scenarios before any scoring run. **Who it is for:** A developer or maintainer who has edited the repo or wants to confirm a clean copy before trusting scores. **The problem:** An edited or inconsistent catalog could silently score a different set of scenarios, which would make results misleading. **The outcome:** A printed count of 94 when the catalog is intact, or an error if the digests fail to verify. ### Steps 1. Clone or open the repo and change into its root directory. 2. Run `PYTHONPATH=src python3 -c "from doc2harness.benchmark.dispatch import active_scenario_ids as a; print(len(a()))"`. 3. Check that the printed output is 94. 4. If the command fails, do not score with this copy until the catalog is restored. ### Prompt to paste into your agent ```text In the repo YuyaoGe/doc2harness, run the Python sanity check that imports active_scenario_ids and prints its length. Report the printed number and any digest error. Verify that the output is exactly 94 before reporting the catalog as intact. ``` ### Caveat The check only confirms the 94-scenario released subset, not the 328-case full suite behind the leaderboard. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/yuyaoge-doc2harness/check-catalog-integrity-before-scoring/ ## How to build a new agent harness and score it with Doc2 Harness Repo: [YuyaoGe/doc2harness](https://github.com/YuyaoGe/doc2harness) | Difficulty: Advanced | Time: 1 hour | Basis: From the README Use the builder image and your own model key to generate a harness, then score it. **Who it is for:** A researcher or developer who wants to see how a chosen model builds a coding agent harness from a development document and a skeleton. **The problem:** Measuring a model on building a full harness, not just writing a patch, requires a repeatable build pipeline and a separate evaluation stage. **The outcome:** A built harness in your output directory, scored against the 94-scenario catalog with per-scenario receipts. ### Steps 1. Run `docker pull ghcr.io/yuyaoge/doc2harness-builder:latest`. 2. Set up a local LiteLLM gateway holding your API key, following src/stage1/README.md. 3. Run `PYTHONPATH=src python3 src/stage1/run_build.py --builder-tag --dry-run` to print the orchestration plan without starting anything. (inferred, check the README) 4. Run `PYTHONPATH=src python3 src/stage1/run_build.py --builder-tag --out-dir results/` for the real build, which the README says takes about 15 minutes. (inferred, check the README) 5. Run the scoring command from the score-existing-agent-harness use case against results/ /agent. ### Prompt to paste into your agent ```text Use the repo YuyaoGe/doc2harness to build a new harness with the builder tag and output to , using the local LiteLLM gateway at . Run the dry-run first, then the real build, and report the output path. Verify the dry-run printed the full plan before starting the real build. ``` ### Caveat The README's build command is truncated in the source, so the exact builder-tag value format should be checked in src/stage1/README.md before running. Risk, credentials: The build needs a local LiteLLM gateway holding your own API key, so keep that key out of shared logs and committed files. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/yuyaoge-doc2harness/build-and-score-new-harness/