DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Jevbench

JevBench is a benchmark and leaderboard for AI decision models that measures typed decision accuracy, calibration, latency and cost.

Trended earlier

fstandhartinger/jevbench on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

3 use cases

  • Shortlist decision models from the JevBench board

    Compare open-weights and hosted decision models on accuracy, calibration, speed and cost before you pick one.

    View prompt
    Using the JevBench project at github.com/fstandhartinger/jevbench, read the release manifest at https://benchmarkheaven.com/api/jevbench/manifest and the results JSON at https://benchmarkheaven.com/api/decision-results/jevbench/v1.6.1/json. Return a table of the five best open-weights systems for <target workload> with their Intelligence, Calibration, Speed and Cost values, marking any null value as missing. Verify that the release source_sha256 matches 5cd8c1332226ea9c179883d2285c0d8b8cc593ae38f2645197364a7ddd9c83c1 before you report results.

  • Test a decision model for option-order sensitivity

    Run the public hard items with options in two orders to see whether your model's answers flip.

    View prompt
    Clone github.com/fstandhartinger/jevbench and load datasets/public/hard.jsonl. Run <your decision model> on each item twice, once with the original option order and once with the options reversed, and output a table of items whose chosen answer changed with the total percentage of flips. Verify that the reference runs in results/v1.2/runs/open-alternative-jev/ are used only for comparison and that your report says the public split can be trained on, so these results are not a held-out score.

  • Compare local decision models with stuntdouble

    Use the independent stuntdouble companion to import JevBench public hard items and compare local models.

    View prompt
    Read the stuntdouble project at github.com/ReallyArtificial/stuntdouble and import the public hard items from github.com/fstandhartinger/jevbench (datasets/public/hard.jsonl) to compare <list of local decision models>. Output a table of results per model and a note that these are not official JevBench results. Verify that the import used only the public split.

How to use Jevbench

  1. Check you have what it needs: A web browser to read the public boards and release manifest; Git and Python for the local repo steps; Your decision model reachable from your own machine if you run the option-order test.
  2. Open fstandhartinger/jevbench on GitHub and read the README and license.
  3. Open https://benchmarkheaven.com/jev-models for the open-weights board or https://benchmarkheaven.com/jev-models/api for the hosted API board.
  4. Read the Capability Score and Composite Score definitions, and note the cost cap of USD 0.0646 per 1,000 decisions and the adjusted median-latency cap of 1.23 seconds.
  5. Read the methodology at https://benchmarkheaven.com/jev-models/methodology and the data card at https://benchmarkheaven.com/jev-models/data before trusting a row.
  6. Check the release version and source_sha256 on https://benchmarkheaven.com/api/jevbench/manifest.

These steps come from our top use case, Shortlist decision models from the JevBench board, which also has a copy-paste agent prompt.

Jevbench alternatives · Best AI repos for Research

About this repo

Requirements

  • A web browser to read the public boards and release manifest
  • Git and Python for the local repo steps
  • Your decision model reachable from your own machine if you run the option-order test
Maintain Jevbench? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/fstandhartinger-jevbench.svg)](https://dedupedai.com/use-cases/fstandhartinger-jevbench/?ref=badge)

More badge styles and HTML

Questions about Jevbench

What is Jevbench?

JevBench is a benchmark and leaderboard for AI decision models that measures typed decision accuracy, calibration, latency and cost. It is an open-source project on GitHub: https://github.com/fstandhartinger/jevbench.

Who is Jevbench for?

Useful for teams choosing a decision model for a workload and for researchers who need to test how a model handles answer option order.

What can I do with Jevbench?

3 use cases so far, for example: Shortlist decision models from the JevBench board; Test a decision model for option-order sensitivity; Compare local decision models with stuntdouble. Each has steps and a copy-paste agent prompt.

What do I need to run Jevbench?

A web browser to read the public boards and release manifest; Git and Python for the local repo steps; Your decision model reachable from your own machine if you run the option-order test.

What are alternatives to Jevbench?

Other hot AI repos in Research: Scientific Agent Skills, Speaker Mem R1, DSH Council, Evo Bot, AI Engineering Course.

Is Jevbench safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.