# AI use cases for Promptfoo (promptfoo/promptfoo) Repo: [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | Use cases: 3 | Page: https://dedupedai.com/use-cases/promptfoo-promptfoo/ Promptfoo is a CLI and library for evaluating and red-teaming LLM-based apps. It runs automated evaluations of prompts and models, compares providers such as OpenAI, Anthropic, Azure, Bedrock and Ollama side by side, scans your own apps for vulnerabilities, automates checks in CI/CD, reviews pull requests for LLM security issues, and runs evals locally so prompts never leave your machine. It is now part of OpenAI and stays open source under MIT. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Compare prompts and models with automated evals: https://dedupedai.com/use-cases/promptfoo-promptfoo/compare-prompts-and-models-with-evals/ - Red team your own LLM app for vulnerabilities: https://dedupedai.com/use-cases/promptfoo-promptfoo/red-team-your-own-llm-app/ - Block prompt regressions in CI: https://dedupedai.com/use-cases/promptfoo-promptfoo/block-prompt-regressions-in-ci/ ## How to compare prompts and models with automated evals with Promptfoo Repo: [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | Difficulty: Beginner | Time: 20 minutes | Basis: From the README Install Promptfoo, start from the example and run an eval with a results viewer. **Who it is for:** A developer choosing between two prompts or two models for a feature. **The problem:** Picking a prompt by eyeballing a few outputs is slow and misses regressions. **The outcome:** A side-by-side table of outputs and pass or fail results for each prompt and model. ### Steps 1. Install with `npm install -g promptfoo`, or `brew install promptfoo`, or use `npx promptfoo@latest` 2. Create a starter project with `promptfoo init --example getting-started` 3. Set your provider key, for example `export OPENAI_API_KEY=sk-abc123` with your real key 4. Run `cd getting-started` then `promptfoo eval` 5. Open the results with `promptfoo view` ### Prompt to paste into your agent ```text Install promptfoo/promptfoo, set up an eval comparing and on and with , run it and open the viewer. Tell me which combination passes the most tests. ``` ### Caveat Evals run locally, but each test still calls the model provider you configure, which may cost money. Risk, credentials: Keep provider API keys in environment variables and out of committed config files. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/promptfoo-promptfoo/compare-prompts-and-models-with-evals/ ## How to red team your own LLM app for vulnerabilities with Promptfoo Repo: [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Use Promptfoo's red teaming to scan an app you own and get a security vulnerability report. **Who it is for:** A team about to launch a customer-facing chatbot or agent that wants to find weak spots first. **The problem:** Prompt injection and data leaks are easy to miss in normal testing. **The outcome:** A vulnerability report for your own app that shows where it fails and what to fix. ### Steps 1. Install Promptfoo with `npm install -g promptfoo` 2. Follow the Red Teaming guide linked in the README to configure a scan of your app 3. Run the scan only against apps you own or are authorised to test 4. Review the security vulnerability report and fix the failing cases 5. Re-run the scan after each fix ### Prompt to paste into your agent ```text Using promptfoo/promptfoo, follow the Red Teaming guide to scan my own app at , which I own. Summarise the vulnerability report by severity and propose a fix for the top three issues. ``` ### Caveat The README links to the Red Teaming guide for configuration rather than showing it. Risk, authorization: Red teaming sends adversarial inputs; run it only against systems you own or have written permission to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/promptfoo-promptfoo/red-team-your-own-llm-app/ ## How to block prompt regressions in CI with Promptfoo Repo: [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README Run Promptfoo evals in your CI/CD pipeline and use code scanning on pull requests. **Who it is for:** An engineering lead tired of prompt edits silently breaking production behaviour. **The problem:** Prompt and model changes ship without tests, so regressions show up as user complaints. **The outcome:** A pipeline that runs your evals on every change and flags LLM security issues in pull requests. ### Steps 1. Write your eval config locally and confirm `promptfoo eval` passes 2. Add Promptfoo to your pipeline as the CI/CD integration docs describe 3. Turn on code scanning to review pull requests for LLM-related security and compliance issues 4. Share results with your team from the viewer 5. Use caching to keep repeated runs fast ### Prompt to paste into your agent ```text Add promptfoo/promptfoo to my so every pull request runs `promptfoo eval` on and fails if any test fails. Also set up code scanning for LLM security issues and show me the workflow file. ``` ### Caveat CI runs call real model APIs, so budget for the cost per run and cache where you can. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/promptfoo-promptfoo/block-prompt-regressions-in-ci/