DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to audit a model's bug-fix claims against the receipts with Bug Hunt Bench

Check a leaderboard score against the raw run data before you trust it.

Short answer: Open the Method page at bughunt.productcompass.pm/method to read how diffs are graded blind against the withheld answer key; then open the raw data in the results/ folder on GitHub; Find the rows for the model you are checking and compare their run counts and scores with the README table. All steps.

Who it is forA reviewer or buyer who needs evidence behind a claim that one model fixes more bugs than another.

The problemA single number on a leaderboard can be hard to trust without seeing the runs that produced it.

The outcomeA verified view of how a score was produced, using the raw results and the stated grading method.

What you need first

  • A web browser to open the live leaderboard at bughunt.productcompass.pm
  • Access to the results/ folder on GitHub or a local copy of the repo for the raw data

Steps

  1. Open the Method page at bughunt.productcompass.pm/method to read how diffs are graded blind against the withheld answer key.
  2. Open the raw data in the results/ folder on GitHub.
  3. Find the rows for the model you are checking and compare their run counts and scores with the README table.
  4. Note any model whose score comes from few runs, such as one run, before you rely on it.

Prompt for your agent

In the bug-hunt-bench repo, open the results/ folder and find the raw data for <model name>. Report its run count, Fixed /105 score, and per-repo scores, and state whether they match the README leaderboard row. Verify the totals before you report.

Caveat. The answer key is withheld, so you can check the scores against the raw data but not re-grade the diffs yourself.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for phuryn/bug-hunt-bench and "Audit a model's bug-fix claims against the receipts"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer