DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Keystone Bench

Keystone is a Python benchmark that tests whether a clinical chat assistant changes its action when one decisive fact is removed, replaced, or contradicted, built on HealthBench conversations.

Trended earlier

xinxuxin/keystone-bench on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

4 use cases

  • Audit a clinical chat assistant's response to missing evidence

    Run the quick set to see whether a model's medical advice moves when a decisive fact changes.

    View prompt
    Use the keystone-bench repo to run the quick layer against <target model> with a judge from a different vendor, using keystone run --family all --layer quick. Confirm that runs/<run>/REPORT.md exists and check the per-family table, including whether controls pass 35 percent of the run.

  • Find which items a new assistant version fixed or broke

    Compare two saved runs item by item to see exactly what changed between versions.

    View prompt
    In the keystone-bench repo, compare two saved runs with keystone regress runs/<old> runs/<new>. Report the items that regressed and the items that were fixed, and confirm both runs used the same quick layer.

  • Confirm a local rebuild matches the published benchmark

    Check your rebuilt twins against the hashes the repository publishes, file by file.

    View prompt
    Use the keystone-bench package to rebuild the release with keystone build, then run keystone build --check. Report whether every file matches the published hashes and list any file that does not, along with the output of keystone pairs.

  • Measure agreement between two judges on the same replies

    Compute Cohen's kappa between two action judges to see whether their verdicts agree.

    View prompt
    In the keystone-bench repo, run keystone compare-judges on two records.jsonl files that classify the same replies with different judges. Report the Cohen's kappa and name the judge pair used.

How to use Keystone Bench

  1. Check you have what it needs: Python with pip to install keystone-bench from PyPI; An API key for an OpenAI-compatible endpoint, such as OpenRouter, to run models and a judge from a different vendor; Network access so that keystone build can fetch HealthBench from OpenAI's public copy and the label files.
  2. Open xinxuxin/keystone-bench on GitHub and read the README and license.
  3. pip install keystone-bench
  4. keystone build
  5. keystone estimate --family all --rubric
  6. export OPENROUTER_API_KEY=...

These steps come from our top use case, Audit a clinical chat assistant's response to missing evidence, which also has a copy-paste agent prompt.

Keystone Bench alternatives · Best AI repos for Research

About this repo

  • Who it is forBuilt for AI safety and evaluation researchers, clinical AI teams, and rubric authors who need to check whether a model's medical advice follows the evidence in the message.
  • Why it is hot
  • LanguagePython
  • Sourcegithub.com/xinxuxin/keystone-bench · Back to Hot repos

Requirements

  • Python with pip to install keystone-bench from PyPI
  • An API key for an OpenAI-compatible endpoint, such as OpenRouter, to run models and a judge from a different vendor
  • Network access so that keystone build can fetch HealthBench from OpenAI's public copy and the label files
  • Enough disk space and time for the local rebuild of the twins, since the release ships span references rather than benchmark prose
Maintain Keystone Bench? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/xinxuxin-keystone-bench.svg)](https://dedupedai.com/use-cases/xinxuxin-keystone-bench/?ref=badge)

More badge styles and HTML

Questions about Keystone Bench

What is Keystone Bench?

Keystone is a Python benchmark that tests whether a clinical chat assistant changes its action when one decisive fact is removed, replaced, or contradicted, built on HealthBench conversations. It is an open-source project on GitHub: https://github.com/xinxuxin/keystone-bench.

Who is Keystone Bench for?

Built for AI safety and evaluation researchers, clinical AI teams, and rubric authors who need to check whether a model's medical advice follows the evidence in the message.

What can I do with Keystone Bench?

4 use cases so far, for example: Audit a clinical chat assistant's response to missing evidence; Find which items a new assistant version fixed or broke; Confirm a local rebuild matches the published benchmark. Each has steps and a copy-paste agent prompt.

What do I need to run Keystone Bench?

Python with pip to install keystone-bench from PyPI; An API key for an OpenAI-compatible endpoint, such as OpenRouter, to run models and a judge from a different vendor; Network access so that keystone build can fetch HealthBench from OpenAI's public copy and the label files; Enough disk space and time for the local rebuild of the twins, since the release ships span references rather than benchmark prose.

What are alternatives to Keystone Bench?

Other hot AI repos in Research: Scientific Agent Skills, Speaker Mem R1, DSH Council, Evo Bot, AI Engineering Course.

Is Keystone Bench safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.