DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to compare prompts and models with automated evals with Promptfoo

Install Promptfoo, start from the example and run an eval with a results viewer.

Short answer: Install with npm install -g promptfoo, or brew install promptfoo, or use npx promptfoo@latest; then create a starter project with promptfoo init --example getting-started; set your provider key, for example export OPENAI_API_KEY=sk-abc123 with your real key. All steps.

Credentials. Keep provider API keys in environment variables and out of committed config files.

Who it is forA developer choosing between two prompts or two models for a feature.

The problemPicking a prompt by eyeballing a few outputs is slow and misses regressions.

The outcomeA side-by-side table of outputs and pass or fail results for each prompt and model.

What you need first

  • Node.js for npm, or Homebrew, or pip; npx works without installing
  • API keys for the LLM providers you test, set as environment variables
  • Only run red teaming against apps you own or are authorised to test

Steps

  1. Install with npm install -g promptfoo, or brew install promptfoo, or use npx promptfoo@latest
  2. Create a starter project with promptfoo init --example getting-started
  3. Set your provider key, for example export OPENAI_API_KEY=sk-abc123 with your real key
  4. Run cd getting-started then promptfoo eval
  5. Open the results with promptfoo view

Prompt for your agent

Install promptfoo/promptfoo, set up an eval comparing <prompt A> and <prompt B> on <model A> and <model B> with <test cases>, run it and open the viewer. Tell me which combination passes the most tests.

Caveat. Evals run locally, but each test still calls the model provider you configure, which may cost money.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for promptfoo/promptfoo and "Compare prompts and models with automated evals"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer