# AI use cases for Eval Forge (jsdhwfmax/EvalForge) Repo: [jsdhwfmax/EvalForge](https://github.com/jsdhwfmax/EvalForge) | Use cases: 4 | Page: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/ EvalForge is a Python tool that turns AI evaluation results into portable evidence, applies versioned release policies, and writes JSON, JUnit, SARIF, and Markdown reports for CI. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Block an AI release when eval scores regress: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/gate-ai-release-in-ci/ - Gate promptfoo eval results before a release: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/import-promptfoo-results-into-gate/ - Enforce thresholds on Ragas or DeepEval scores: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/gate-ragas-or-deepeval-scores/ - Compare two RAG retrieval setups without a model key: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/compare-rag-configs-offline/ ## How to block an AI release when eval scores regress with Eval Forge Repo: [jsdhwfmax/EvalForge](https://github.com/jsdhwfmax/EvalForge) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Run a baseline-versus-candidate gate that exits non-zero and writes CI-readable reports. **Who it is for:** A maintainer of a RAG app or AI assistant who needs a clear pass or fail check on each pull request before a model, prompt, or retrieval change ships. **The problem:** Metric numbers exist, but there is no shared answer to what changed, which limits were enforced, and where the evidence is. **The outcome:** A gate result with JSON, JUnit, SARIF, and Markdown reports, and a non-zero exit code when a required check fails. ### Steps 1. Install the gate with `pip install evalforge-ci`. 2. Write a policy file with absolute thresholds and baseline deltas, or start from `examples/quality_policy.json`. 3. Run `evalforge gate examples/candidate_summary.json --policy examples/quality_policy.json --baseline examples/baseline_summary.json --json build/evalforge-report.json --junit build/evalforge-junit.xml --sarif build/evalforge.sarif --markdown build/evalforge-summary.md`. (inferred, check the README) 4. Check the exit code: 0 for pass, 1 for a failed or error check, 2 for invalid input or policy. 5. Add the reusable Action with `uses: jsdhwfmax/EvalForge@v0.6.0` and set `candidate`, `baseline`, and `policy` inputs. ### Prompt to paste into your agent ```text Using the EvalForge repo at https://github.com/jsdhwfmax/EvalForge, install evalforge-ci and run `evalforge gate --policy --baseline --json build/evalforge-report.json`. Expected output is a pass or fail verdict and the JSON report; verify the exit code is 0 or 1 and not 2. ``` ### Caveat Eval Forge is v0.6.0 alpha, and the gate only judges the numbers you feed it, so the quality of the check depends on the evaluator that produced them. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/gate-ai-release-in-ci/ ## How to gate promptfoo eval results before a release with Eval Forge Repo: [jsdhwfmax/EvalForge](https://github.com/jsdhwfmax/EvalForge) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Convert promptfoo JSON output into a portable artifact and run a policy gate on it. **Who it is for:** A developer who already runs promptfoo evaluations and wants a release decision in CI without sending prompts or traces anywhere. **The problem:** promptfoo output uses its own result shape, so it cannot be compared with a baseline or fed to a CI policy directly. **The outcome:** A portable Eval Forge artifact built from promptfoo results, checked by a policy file that does not copy prompts, responses, or traces. ### Steps 1. Install the gate with `pip install evalforge-ci`. 2. Run `promptfoo eval --output build/promptfoo-results.json`. 3. Run `evalforge import promptfoo build/promptfoo-results.json --output build/evalforge-artifact.json --source-revision "$GITHUB_SHA"`. (inferred, check the README) 4. Run `evalforge gate build/evalforge-artifact.json --policy examples/promptfoo_policy.json`. (inferred, check the README) ### Prompt to paste into your agent ```text In the EvalForge repo (https://github.com/jsdhwfmax/EvalForge), convert a promptfoo results file at with `evalforge import promptfoo --output build/evalforge-artifact.json`, then run `evalforge gate build/evalforge-artifact.json --policy examples/promptfoo_policy.json`. Expected output is the gate verdict; verify the adapter accepted results schema 3. ``` ### Caveat The promptfoo adapter is verified against promptfoo 0.122.2 and requires results schema 3, so other promptfoo versions may be rejected. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/import-promptfoo-results-into-gate/ ## How to enforce thresholds on Ragas or DeepEval scores with Eval Forge Repo: [jsdhwfmax/EvalForge](https://github.com/jsdhwfmax/EvalForge) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Import selected Ragas or DeepEval metrics and apply a policy gate to them. **Who it is for:** A RAG team that already scores answers with Ragas or DeepEval and wants one policy check across both frameworks in CI. **The problem:** Framework score files mix metrics, sample text, and traces, and the score directions can differ between versions. **The outcome:** A portable artifact that carries only the chosen score columns, checked against a policy, with missing or errored evidence failing the import. ### Steps 1. Install the gate with `pip install evalforge-ci`. 2. Export Ragas results with `result.to_pandas().to_json(..., orient="records")` and choose the score columns you need. 3. Run `evalforge import ragas --producer-version --metric faithfulness --metric answer_relevancy --dataset-fingerprint --metric-version --output build/ragas.json`. (inferred, check the README) 4. Run `evalforge gate build/ragas.json --policy examples/ragas_policy.json`. 5. For DeepEval, run `evalforge import deepeval --producer-version 4.2.2 --output build/deepeval.json`, then gate it with `evalforge gate build/deepeval.json --policy examples/deepeval_policy.json`. (inferred, check the README) ### Prompt to paste into your agent ```text Using https://github.com/jsdhwfmax/EvalForge, import a Ragas export at with `evalforge import ragas` using explicit --metric flags, then run `evalforge gate` on the output with examples/ragas_policy.json. Expected result is a gate verdict built only from the selected score columns; verify that no sample text appears in the output artifact. ``` ### Caveat The repo's Ragas and DeepEval fixtures contain synthetic scores that show interoperability, not real model quality, and DeepEval support is pinned to versions 3.8.1 and 4.2.2. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/gate-ragas-or-deepeval-scores/ ## How to compare two RAG retrieval setups without a model key with Eval Forge Repo: [jsdhwfmax/EvalForge](https://github.com/jsdhwfmax/EvalForge) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README Run baseline and candidate RAG configurations offline and review the deltas in a dashboard. **Who it is for:** A team tuning retrieval for a RAG app that wants to know whether a hybrid setup beats a baseline before choosing it, without a hosted model account. **The problem:** Changes to retrieval settings look better or worse by feel, and comparing them on different test sets gives misleading results. **The outcome:** Two exported experiment summaries with matching dataset fingerprints, a gate verdict on the deltas, and a dashboard comparison. ### Steps 1. Run `docker compose up --build` from a clone of the repo, then open the dashboard at the address shown in the README. 2. Run `evalforge run baseline_top1 --name "Baseline" --output build/baseline.json`. 3. Run `evalforge run hybrid_top3 --name "Candidate" --output build/candidate.json`. 4. Run `evalforge check hybrid_top3 --name "PR candidate" --report-dir artifacts` to apply the release gate and write reports. ### Prompt to paste into your agent ```text In the EvalForge repo (https://github.com/jsdhwfmax/EvalForge), start the stack with `docker compose up --build`, then run `evalforge run baseline_top1 --name "Baseline" --output build/baseline.json` and `evalforge run hybrid_top3 --name "Candidate" --output build/candidate.json`. Expected output is two experiment JSON files; verify both share the same dataset fingerprint before comparing them. ``` ### Caveat The offline evaluator is an extractive baseline with hashing embeddings, so it demonstrates the workflow but does not measure how a hosted model would perform. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jsdhwfmax-evalforge/compare-rag-configs-offline/