DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to measure how often answers flip when options are reordered with Open Jev Verdict 2 0

Test whether a decision model gives the same answer when option order changes.

Who it is forA machine learning engineer who must justify a decision classifier to a risk or compliance team and needs evidence that prompt order does not drive results.

The problemAutoregressive models can change answers when option positions are shuffled, and a team needs a measured flip rate before trusting the output.

The outcomeA flip rate for your own shuffled option sets, compared against the 4.76% flip rate and the 7.41% flip rate for Kev-0.5B that the README reports.

What you need first

  • Python with the transformers library installed
  • Access to the model files; the Verdict 2.0 weights are tracked via Git LFS pointers in artifacts/verdict2-base/model.pt
  • Chrome or Edge for the in-browser WebGPU playground
  • A GPU is optional; the README reports training on a 6 GB laptop GPU

Steps

  1. Clone the repository with git clone https://github.com/Heman10x-NGU/openJev-verdict-2.0. (inferred, check the README)
  2. Read the Symmetric Permutation-KL section and the flip rate definitions in the README.
  3. Build a set of decisions where you shuffle the option order for each item, keeping the same correct answer.
  4. Run python verdict2/evaluate.py and collect the predicted option for each shuffled version. (inferred, check the README)
  5. Count the items where the chosen option changes after reordering and divide by the total number of items.

Prompt for your agent

In the repo openJev-verdict-2.0, read the Symmetric Permutation-KL section and the evaluation script at verdict2/evaluate.py. Write a script that shuffles option order for each decision in <decision file> and reports the percentage of answers that flip. Expected output: a flip rate percentage, and verify that the script reports the same option when the input order is unchanged.

Caveat. The README reports the flip rate on its own 2,918 perturbed decisions, so your own set may give different numbers and the flip rate does not measure accuracy.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page