# AI use cases for Jevbench (fstandhartinger/jevbench)

Repo: [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) | Use cases: 3 | Page: https://dedupedai.com/use-cases/fstandhartinger-jevbench/

JevBench is a benchmark and leaderboard for AI decision models that measures typed decision accuracy, calibration, latency and cost.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Shortlist decision models from the JevBench board: https://dedupedai.com/use-cases/fstandhartinger-jevbench/shortlist-decision-models-from-jevbench-board/
- Test a decision model for option-order sensitivity: https://dedupedai.com/use-cases/fstandhartinger-jevbench/test-option-order-sensitivity-before-deploy/
- Compare local decision models with stuntdouble: https://dedupedai.com/use-cases/fstandhartinger-jevbench/compare-local-models-with-stuntdouble-import/

## How to shortlist decision models from the JevBench board with Jevbench

Repo: [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README

Compare open-weights and hosted decision models on accuracy, calibration, speed and cost before you pick one.

**Who it is for:** A product or operations lead who needs a decision model for a typed decision workflow and wants published measurements before testing anything.

**The problem:** Choosing a decision model from vendor claims is hard to compare, and a single number hides accuracy, calibration, latency and cost trade-offs.

**The outcome:** A short list of candidate systems with their Intelligence, Calibration, Speed and Cost values, plus a clear note of where measurements are missing.

### Steps

1. Open https://benchmarkheaven.com/jev-models for the open-weights board or https://benchmarkheaven.com/jev-models/api for the hosted API board.
2. Read the Capability Score and Composite Score definitions, and note the cost cap of USD 0.0646 per 1,000 decisions and the adjusted median-latency cap of 1.23 seconds.
3. Read the methodology at https://benchmarkheaven.com/jev-models/methodology and the data card at https://benchmarkheaven.com/jev-models/data before trusting a row.
4. Check the release version and source_sha256 on https://benchmarkheaven.com/api/jevbench/manifest.
5. Shortlist systems, and treat any null measurement as missing data, never as zero.

### Prompt to paste into your agent

```text
Using the JevBench project at github.com/fstandhartinger/jevbench, read the release manifest at https://benchmarkheaven.com/api/jevbench/manifest and the results JSON at https://benchmarkheaven.com/api/decision-results/jevbench/v1.6.1/json. Return a table of the five best open-weights systems for <target workload> with their Intelligence, Calibration, Speed and Cost values, marking any null value as missing. Verify that the release source_sha256 matches 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1 before you report results.
```

### Caveat

A leaderboard score does not establish reliability on your workload, and scores from different method versions should not be compared as the same experiment.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/fstandhartinger-jevbench/shortlist-decision-models-from-jevbench-board/

## How to test a decision model for option-order sensitivity with Jevbench

Repo: [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Run the public hard items with options in two orders to see whether your model's answers flip.

**Who it is for:** A machine learning engineer who is about to ship a small decision model and needs to know if its answers depend on how options are listed.

**The problem:** The README reports that one small model scored 21 % instead of 72 % on answer-judging items when option order was reversed, so order can silently change results.

**The outcome:** A comparison of answers under original and reversed option order, with a list of items where the model's choice changed.

### Steps

1. Clone the repo with `git clone https://github.com/fstandhartinger/jevbench` and open the folder. (inferred, check the README)
2. Open datasets/public/hard.jsonl, which holds 111 public hard items.
3. Run your decision model on every item twice: once with the options in the original order and once with the options reversed.
4. Compare the two sets of answers and count the items where the chosen answer changed.
5. Compare your results with the reference runs in results/v1.2/runs/open-alternative-jev/ and the analysis in issue #40.

### Prompt to paste into your agent

```text
Clone github.com/fstandhartinger/jevbench and load datasets/public/hard.jsonl. Run <your decision model> on each item twice, once with the original option order and once with the options reversed, and output a table of items whose chosen answer changed with the total percentage of flips. Verify that the reference runs in results/v1.2/runs/open-alternative-jev/ are used only for comparison and that your report says the public split can be trained on, so these results are not a held-out score.
```

### Caveat

The public items can be trained on or selected against, so a flip test on them shows sensitivity but does not give a held-out accuracy score.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/fstandhartinger-jevbench/test-option-order-sensitivity-before-deploy/

## How to compare local decision models with stuntdouble with Jevbench

Repo: [fstandhartinger/jevbench](https://github.com/fstandhartinger/jevbench) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Use the independent stuntdouble companion to import JevBench public hard items and compare local models.

**Who it is for:** A developer running local decision models who wants a side-by-side comparison on the same public items.

**The problem:** Local model tests are hard to repeat and compare when every setup uses different items and tooling.

**The outcome:** A replay of the same public JevBench hard items across your local decision models, with a report you can rerun.

### Steps

1. Open the stuntdouble project at https://github.com/ReallyArtificial/stuntdouble and read its setup notes.
2. Confirm that stuntdouble can import JevBench's public hard items, as the README describes.
3. Import the public hard items from the JevBench repo into stuntdouble.
4. Run each local decision model through the import and record the results for each one.
5. Write up the comparison and label it as your own result, since stuntdouble is not a JevBench entrant and its example report is not an official JevBench result.

### Prompt to paste into your agent

```text
Read the stuntdouble project at github.com/ReallyArtificial/stuntdouble and import the public hard items from github.com/fstandhartinger/jevbench (datasets/public/hard.jsonl) to compare <list of local decision models>. Output a table of results per model and a note that these are not official JevBench results. Verify that the import used only the public split.
```

### Caveat

stuntdouble is a third-party tool, and its example report is not an official JevBench result, so cite it only as your own comparison.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/fstandhartinger-jevbench/compare-local-models-with-stuntdouble-import/