DEDUPED Hot AI repos, turned into use cases Get 5 a day

What you can do with Eval Forge

EvalForge is a Python tool that turns AI evaluation results into portable evidence, applies versioned release policies, and writes JSON, JUnit, SARIF, and Markdown reports for CI.

Trended earlier

jsdhwfmax/EvalForge on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

4 use cases

  • Block an AI release when eval scores regress

    Run a baseline-versus-candidate gate that exits non-zero and writes CI-readable reports.

    View prompt
    Using the EvalForge repo at https://github.com/jsdhwfmax/EvalForge, install evalforge-ci and run `evalforge gate <candidate-summary-path> --policy <policy-path> --baseline <baseline-path> --json build/evalforge-report.json`. Expected output is a pass or fail verdict and the JSON report; verify the exit code is 0 or 1 and not 2.

  • Gate promptfoo eval results before a release

    Convert promptfoo JSON output into a portable artifact and run a policy gate on it.

    View prompt
    In the EvalForge repo (https://github.com/jsdhwfmax/EvalForge), convert a promptfoo results file at <promptfoo-results-path> with `evalforge import promptfoo <promptfoo-results-path> --output build/evalforge-artifact.json`, then run `evalforge gate build/evalforge-artifact.json --policy examples/promptfoo_policy.json`. Expected output is the gate verdict; verify the adapter accepted results schema 3.

  • Enforce thresholds on Ragas or DeepEval scores

    Import selected Ragas or DeepEval metrics and apply a policy gate to them.

    View prompt
    Using https://github.com/jsdhwfmax/EvalForge, import a Ragas export at <ragas-records-path> with `evalforge import ragas` using explicit --metric flags, then run `evalforge gate` on the output with examples/ragas_policy.json. Expected result is a gate verdict built only from the selected score columns; verify that no sample text appears in the output artifact.

  • Compare two RAG retrieval setups without a model key

    Run baseline and candidate RAG configurations offline and review the deltas in a dashboard.

    View prompt
    In the EvalForge repo (https://github.com/jsdhwfmax/EvalForge), start the stack with `docker compose up --build`, then run `evalforge run baseline_top1 --name "Baseline" --output build/baseline.json` and `evalforge run hybrid_top3 --name "Candidate" --output build/candidate.json`. Expected output is two experiment JSON files; verify both share the same dataset fingerprint before comparing them.

How to use Eval Forge

  1. Check you have what it needs: Python 3 with pip and a virtual environment; A committed evaluation artifact, or results from promptfoo, Ragas, or DeepEval; Git, and a GitHub repository if you want the composite Action.
  2. Open jsdhwfmax/EvalForge on GitHub and read the README and license.
  3. Install the gate with pip install evalforge-ci.
  4. Write a policy file with absolute thresholds and baseline deltas, or start from examples/quality_policy.json.
  5. Run evalforge gate examples/candidate_summary.json --policy examples/quality_policy.json --baseline examples/baseline_summary.json --json build/evalforge-report.json --junit build/evalforge-junit.xml --sarif build/evalforge.sarif --markdown build/evalforge-summary.md. (inferred, check the README)
  6. Check the exit code: 0 for pass, 1 for a failed or error check, 2 for invalid input or policy.

These steps come from our top use case, Block an AI release when eval scores regress, which also has a copy-paste agent prompt.

Eval Forge alternatives · Best AI repos for Software and DevOps

About this repo

Requirements

  • Python 3 with pip and a virtual environment
  • A committed evaluation artifact, or results from promptfoo, Ragas, or DeepEval
  • Git, and a GitHub repository if you want the composite Action
  • Docker, only if you use the Compose path
Maintain Eval Forge? Add the Featured on Deduped badge

Paste this into your README:

[![Featured on Deduped](https://dedupedai.com/badge/jsdhwfmax-evalforge.svg)](https://dedupedai.com/use-cases/jsdhwfmax-evalforge/?ref=badge)

More badge styles and HTML

Questions about Eval Forge

What is Eval Forge?

EvalForge is a Python tool that turns AI evaluation results into portable evidence, applies versioned release policies, and writes JSON, JUnit, SARIF, and Markdown reports for CI. It is an open-source project on GitHub: https://github.com/jsdhwfmax/EvalForge.

Who is Eval Forge for?

Built for maintainers and CI owners of RAG apps or AI assistants who need a pass or fail release gate on evaluation results.

What can I do with Eval Forge?

4 use cases so far, for example: Block an AI release when eval scores regress; Gate promptfoo eval results before a release; Enforce thresholds on Ragas or DeepEval scores. Each has steps and a copy-paste agent prompt.

What do I need to run Eval Forge?

Python 3 with pip and a virtual environment; A committed evaluation artifact, or results from promptfoo, Ragas, or DeepEval; Git, and a GitHub repository if you want the composite Action; Docker, only if you use the Compose path.

What are alternatives to Eval Forge?

Other hot AI repos in Software and DevOps: Harness2, iFixAi, Orca, Diagram Design, AI Memory.

Is Eval Forge safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.