# AI use cases for Keystone Bench (xinxuxin/keystone-bench)

Repo: [xinxuxin/keystone-bench](https://github.com/xinxuxin/keystone-bench) | Use cases: 4 | Page: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/

Keystone is a Python benchmark that tests whether a clinical chat assistant changes its action when one decisive fact is removed, replaced, or contradicted, built on HealthBench conversations.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Audit a clinical chat assistant's response to missing evidence: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/audit-assistant-decision-evidence/
- Find which items a new assistant version fixed or broke: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/check-what-a-new-version-regressed/
- Confirm a local rebuild matches the published benchmark: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/verify-benchmark-rebuild-hashes/
- Measure agreement between two judges on the same replies: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/test-judge-agreement-on-replies/

## How to audit a clinical chat assistant's response to missing evidence with Keystone Bench

Repo: [xinxuxin/keystone-bench](https://github.com/xinxuxin/keystone-bench) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Run the quick set to see whether a model's medical advice moves when a decisive fact changes.

**Who it is for:** An AI safety or clinical evaluation researcher who needs to test a chat assistant before it answers patient-style questions.

**The problem:** A physician-written rubric scores a reply to the message the physician saw, so it says nothing about what the assistant does when a decisive fact is missing, replaced, or contradicted.

**The outcome:** A per-family report with rates for how often the assistant's action follows the evidence on the twins, with the negative controls reported separately.

### Steps

1. `pip install keystone-bench`
2. `keystone build`
3. `keystone estimate --family all --rubric`
4. `export OPENROUTER_API_KEY=...`
5. `keystone run --family all --layer quick --model openrouter/openai/gpt-5.6-terra --judge openrouter/anthropic/claude-sonnet-5`

### Prompt to paste into your agent

```text
Use the keystone-bench repo to run the quick layer against <target model> with a judge from a different vendor, using keystone run --family all --layer quick. Confirm that runs/<run>/REPORT.md exists and check the per-family table, including whether controls pass 35 percent of the run.
```

### Caveat

Results describe the assistant on these twins only, and the action judge must come from a different vendor than the model under test.

Risk, credentials: The run needs an API key for an OpenAI-compatible endpoint, so store it in an environment variable and do not commit it.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/audit-assistant-decision-evidence/

## How to find which items a new assistant version fixed or broke with Keystone Bench

Repo: [xinxuxin/keystone-bench](https://github.com/xinxuxin/keystone-bench) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Compare two saved runs item by item to see exactly what changed between versions.

**Who it is for:** A team shipping updates to a medical chat assistant that needs to know whether a new version got worse on specific decisive-fact cases.

**The problem:** A single aggregate score can hide that a new version fixed some missing-evidence cases while breaking others.

**The outcome:** A list of items that changed verdict between the two runs, so the team can review the regressions directly.

### Steps

1. `pip install keystone-bench`
2. `keystone build`
3. `keystone run --family all --layer quick --model openrouter/openai/gpt-5.6-terra --judge openrouter/anthropic/claude-sonnet-5`
4. Rerun the same command for the new version with a different output folder, so you have two run folders to compare (inferred, check the README)
5. `keystone regress runs/v1 runs/v2`

### Prompt to paste into your agent

```text
In the keystone-bench repo, compare two saved runs with keystone regress runs/<old> runs/<new>. Report the items that regressed and the items that were fixed, and confirm both runs used the same quick layer.
```

### Caveat

Both runs must use the same release and layer for the comparison to be meaningful, and the regression check only covers the twins in those runs.

Risk, credentials: Each run calls a model endpoint, so the API key must be set in the environment before the runs start.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/check-what-a-new-version-regressed/

## How to confirm a local rebuild matches the published benchmark with Keystone Bench

Repo: [xinxuxin/keystone-bench](https://github.com/xinxuxin/keystone-bench) | Difficulty: Beginner | Time: 30 minutes | Basis: From the README

Check your rebuilt twins against the hashes the repository publishes, file by file.

**Who it is for:** A researcher or auditor who wants to cite Keystone results and needs proof that their local copy matches the released data.

**The problem:** The twins are not shipped in the package, so a reader has no direct way to know their local build matches what was evaluated.

**The outcome:** A file-by-file match or mismatch report against the published SHA-256 hashes.

### Steps

1. `pip install keystone-bench`
2. `keystone build`
3. `keystone build --check`
4. `keystone pairs`

### Prompt to paste into your agent

```text
Use the keystone-bench package to rebuild the release with keystone build, then run keystone build --check. Report whether every file matches the published hashes and list any file that does not, along with the output of keystone pairs.
```

### Caveat

The rebuild fetches HealthBench from OpenAI's public copy, so the check only confirms a match if that source is also unchanged.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/verify-benchmark-rebuild-hashes/

## How to measure agreement between two judges on the same replies with Keystone Bench

Repo: [xinxuxin/keystone-bench](https://github.com/xinxuxin/keystone-bench) | Difficulty: Advanced | Time: 1 hour | Basis: From the README

Compute Cohen's kappa between two action judges to see whether their verdicts agree.

**Who it is for:** A rubric or grader author who needs to know whether a second judge reads the same replies the same way before trusting a score.

**The problem:** An action judge's verdicts can depend on which model reads the reply, and a single judge gives no sense of that spread.

**The outcome:** A kappa score showing how far two judges agree on the same set of classified replies.

### Steps

1. `pip install keystone-bench`
2. `keystone run --family all --layer quick --model openrouter/openai/gpt-5.6-terra --judge openrouter/anthropic/claude-sonnet-5`
3. Rerun the same replies with a second judge from another vendor into a separate run folder (inferred, check the README)
4. `keystone compare-judges runs/a/records.jsonl runs/b/records.jsonl` (inferred, check the README)

### Prompt to paste into your agent

```text
In the keystone-bench repo, run keystone compare-judges on two records.jsonl files that classify the same replies with different judges. Report the Cohen's kappa and name the judge pair used.
```

### Caveat

Kappa between two judges shows agreement only; it does not show which judge is correct on a given reply.

Risk, credentials: Running two judges means two model endpoints, so each API key must be set in the environment before the runs.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/xinxuxin-keystone-bench/test-judge-agreement-on-replies/