DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to block prompt regressions in CI with Promptfoo

Run Promptfoo evals in your CI/CD pipeline and use code scanning on pull requests.

Short answer: Write your eval config locally and confirm promptfoo eval passes; then add Promptfoo to your pipeline as the CI/CD integration docs describe; turn on code scanning to review pull requests for LLM-related security and compliance issues. All steps.

Who it is forAn engineering lead tired of prompt edits silently breaking production behaviour.

The problemPrompt and model changes ship without tests, so regressions show up as user complaints.

The outcomeA pipeline that runs your evals on every change and flags LLM security issues in pull requests.

What you need first

  • Node.js for npm, or Homebrew, or pip; npx works without installing
  • API keys for the LLM providers you test, set as environment variables
  • Only run red teaming against apps you own or are authorised to test

Steps

  1. Write your eval config locally and confirm promptfoo eval passes
  2. Add Promptfoo to your pipeline as the CI/CD integration docs describe
  3. Turn on code scanning to review pull requests for LLM-related security and compliance issues
  4. Share results with your team from the viewer
  5. Use caching to keep repeated runs fast

Prompt for your agent

Add promptfoo/promptfoo to my <CI system> so every pull request runs `promptfoo eval` on <config> and fails if any test fails. Also set up code scanning for LLM security issues and show me the workflow file.

Caveat. CI runs call real model APIs, so budget for the cost per run and cache where you can.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for promptfoo/promptfoo and "Block prompt regressions in CI"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer