# AI use cases for Bug Hunt Bench (phuryn/bug-hunt-bench) Repo: [phuryn/bug-hunt-bench](https://github.com/phuryn/bug-hunt-bench) | Use cases: 4 | Page: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/ A benchmark of 105 real bugs in two production codebases that grades frontier AI coding models, each run in its own agentic CLI, with blind grading against a withheld answer key. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Choose a coding model for bug fixing by score and cost: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/pick-coding-model-for-bug-fixing/ - Compare one model across different coding CLIs: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/compare-one-model-across-coding-clis/ - Study the 25 bugs no model has fixed: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/study-the-25-bugs-no-model-fixed/ - Audit a model's bug-fix claims against the receipts: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/audit-model-fix-claims-against-receipts/ ## How to choose a coding model for bug fixing by score and cost with Bug Hunt Bench Repo: [phuryn/bug-hunt-bench](https://github.com/phuryn/bug-hunt-bench) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README Rank models by planted bugs fixed, per repo and per dollar, before you pick one for fixes. **Who it is for:** An engineering lead choosing which AI coding model and CLI to pay for when the team's bug backlog keeps growing. **The problem:** Vendor claims about coding ability are hard to compare, and a model that scores well on one task can cost far more than a close rival. **The outcome:** A shortlist of models with their Fixed /105 score, per-repo scores, and cost, so the choice rests on measured bug fixes. ### Steps 1. Open the live board at bughunt.productcompass.pm. 2. Read the Fixed /105 column and the Repo 1 /45 and Repo 2 /60 columns to see where each model does well. 3. Compare the Cost column for the models that score within a few points of the top. 4. Open the Method page at bughunt.productcompass.pm/method and read the definitions and caveats before you decide. ### Prompt to paste into your agent ```text Using the bug-hunt-bench leaderboard at bughunt.productcompass.pm, list the top five models by Fixed /105 along with their Repo 1 and Repo 2 scores and Cost for . Output a short table. Verify that each score matches the row in the README table before reporting it. ``` ### Caveat Scores count only the 105 planted bugs in this bench, so a high score says little about bugs outside it. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/pick-coding-model-for-bug-fixing/ ## How to compare one model across different coding CLIs with Bug Hunt Bench Repo: [phuryn/bug-hunt-bench](https://github.com/phuryn/bug-hunt-bench) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README See whether the tool wrapped around a model changes its bug-fixing score. **Who it is for:** A developer who runs the same model in several coding tools and sees different results from each. **The problem:** A model's score can reflect the harness it runs in, so a score from one tool may not carry over to another. **The outcome:** A clear view of how the same model scores across harnesses and effort levels, which helps decide which tool to use. ### Steps 1. Open the leaderboard at bughunt.productcompass.pm. 2. Find one model in the table, for example DeepSeek V4.1 Flash 1017, and note its Harness and Effort values. 3. Compare its Fixed /105 scores across harnesses: DeepSeek Harness, Pi, Claude Code and OpenCode at max effort score 42.7, 42.2, 39.1 and 37.6. 4. Check the Effort column to see whether the same harness scores differently at high, low or max effort. ### Prompt to paste into your agent ```text Using the bug-hunt-bench README table, list every row for with its Harness, Effort, and Fixed /105 score, sorted by score. Output a table. Verify that each row's harness and effort match the source table. ``` ### Caveat Run counts differ across rows, so a difference of one or two points between harnesses may be noise. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/compare-one-model-across-coding-clis/ ## How to study the 25 bugs no model has fixed with Bug Hunt Bench Repo: [phuryn/bug-hunt-bench](https://github.com/phuryn/bug-hunt-bench) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Look at the bugs that every model missed to learn where current tools fail. **Who it is for:** A researcher or engineer who wants to understand the hardest bugs in the bench and why models miss them. **The problem:** Aggregate scores hide which specific bugs stay unfixed, and those are the cases that show real limits. **The outcome:** A list of the bugs no model fixed, with context from the findings by wave, to guide further study. ### Steps 1. Note the README fact that 25 of 105 bugs have never been fixed by any model. 2. Open the raw data in the results/ folder on GitHub. 3. Open results/waves.md to read the findings by wave. 4. Match the unfixed bugs against the wave findings and write down the patterns you see. ### Prompt to paste into your agent ```text In the bug-hunt-bench repo, open results/waves.md and the raw data in results/, then list the bugs that no model fixed, grouped by repo. Output a short report. Verify that the count of unfixed bugs matches the 25 of 105 stated in the README. ``` ### Caveat The unfixed list reflects only the models and runs in this bench, so other models may fix some of these bugs. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/study-the-25-bugs-no-model-fixed/ ## How to audit a model's bug-fix claims against the receipts with Bug Hunt Bench Repo: [phuryn/bug-hunt-bench](https://github.com/phuryn/bug-hunt-bench) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Check a leaderboard score against the raw run data before you trust it. **Who it is for:** A reviewer or buyer who needs evidence behind a claim that one model fixes more bugs than another. **The problem:** A single number on a leaderboard can be hard to trust without seeing the runs that produced it. **The outcome:** A verified view of how a score was produced, using the raw results and the stated grading method. ### Steps 1. Open the Method page at bughunt.productcompass.pm/method to read how diffs are graded blind against the withheld answer key. 2. Open the raw data in the results/ folder on GitHub. 3. Find the rows for the model you are checking and compare their run counts and scores with the README table. 4. Note any model whose score comes from few runs, such as one run, before you rely on it. ### Prompt to paste into your agent ```text In the bug-hunt-bench repo, open the results/ folder and find the raw data for . Report its run count, Fixed /105 score, and per-repo scores, and state whether they match the README leaderboard row. Verify the totals before you report. ``` ### Caveat The answer key is withheld, so you can check the scores against the raw data but not re-grade the diffs yourself. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/phuryn-bug-hunt-bench/audit-model-fix-claims-against-receipts/