DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to see a forecaster's calibration gap in four seconds with Brier

Run the bundled demo and score it to see how stated confidence compares with actual hit rate.

Short answer: Open a terminal in an empty folder with Node.js installed; then run npx brier demo && npx brier score; read the row for each forecaster and compare the said column with the right column. All steps.

  • Top pick
  • Research
  • Intermediate
  • 15 minutes
  • From the README
  • Any agent

Who it is forA developer or analyst who wants to see how a forecaster's stated probabilities line up with real outcomes before trusting it.

The problemForecasts are usually judged from memory, and a confident model can look right while being consistently overconfident.

The outcomeA scoreboard showing each forecaster's said percentage, how often it was right, the gap between them, and its skill against the base rate.

What you need first

  • A terminal with Node.js and npx, since the README's commands run through npx
  • An API key in a .env file, only if you want your own models on the board
  • A wallet you control, only if you want to anchor the head hash on Robinhood Chain

Steps

  1. Open a terminal in an empty folder with Node.js installed.
  2. Run npx brier demo && npx brier score.
  3. Read the row for each forecaster and compare the said column with the right column.
  4. Check the always-yes row to see how a trivial forecaster scores on the same question set.

Prompt for your agent

Use the Noisyxl/brier repo. Run `npx brier demo && npx brier score` in a clean directory and report the said, right, gap and skill columns for each forecaster. Verify that the output includes an always-yes row before summarizing.

Caveat. The demo shows the ledger's own bundled questions and forecasters, so its numbers describe that set and do not generalize to your domain.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Noisyxl/brier and "See a forecaster's calibration gap in four seconds"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer