DEDUPED Hot AI repos, turned into use cases Get the Monday 5

What you can do with Jevals

jevals is a Python library for running agent and quality evals and guardrails by sending all checks for a trace to a decision model in one request.

Trended earlier

openlayer-ai/jevals on GitHub

Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk.

What do you want to do?

5 use cases

  • Flag agent answers that invent facts the tools never returned

    Score each claim in an agent's final answer against its tool results in one request.

    View prompt
    Use the jevals repo to evaluate the agent trace in <trace_file>. Call evaluate() with UsedToolResult() and Grounded() on its messages and tools. Report the grounded score and every claim with a low support probability. Verify the report lists the claims in the order they appear in the final answer.

  • Catch tool results that try to give the agent orders

    Check every tool result for indirect prompt injection and PHI in the same request as your other evals.

    View prompt
    In the jevals repo, run IndirectInjection() and PHI() from jevals.security against the traces in <traces_file>. Report each trace that fails indirect_injection, with the tool result text that triggered it. Verify that the PHI check ran with the jevals[pii] extra installed.

  • Block agent replies that fail checks before they reach the user

    Run evals inside an async agent loop with aevaluate() or a gate's acheck().

    View prompt
    Wire jevals into the async agent loop in <agent_file>. Use aevaluate() with StayedInScope() and AnswerRelevancy() on each reply before it is returned. Report the latency added per reply. Verify that a reply that fails either check is not sent to the user.

  • Let Cursor or Claude Code write and run agent evals for you

    Install the Jevals MCP server so your coding agent can run and author evals.

    View prompt
    Using the jevals MCP server, write an eval for <agent_repo> that checks whether the agent used its tool results. Run it on one trace from <traces_file>. Report the score and the trace id. Verify the eval runs without a syntax error before you report.

  • Run agent evals on your own Mac with no API calls

    Serve Kev or use Laya on Apple Silicon so evals run locally.

    View prompt
    Set up jevals to run locally on this Mac using Kev or Laya. Serve the decision model, run evaluate() with ToolChoice() and Grounded() on <trace_file>, and report the local latency. Verify that no request leaves the machine by checking the backend URL is localhost.

How to use Jevals

  1. Check you have what it needs: Python 3.10 or newer; An API key for one backend: AI_GATEWAY_API_KEY (Jev through Vercel AI Gateway), TYPESAFE_API_KEY, or OPENROUTER_API_KEY; A trace to evaluate: a messages list in the OpenAI chat format plus the tools you gave the model.
  2. Open openlayer-ai/jevals on GitHub and read the README and license.
  3. pip install jevals
  4. export AI_GATEWAY_API_KEY=... (TYPESAFE_API_KEY or OPENROUTER_API_KEY also work)
  5. Save a script that calls evaluate({"messages": messages, "tools": tools}, [UsedToolResult(), Grounded()]) with from jevals.agent import UsedToolResult, Grounded (inferred, check the README)
  6. Run the script on one saved trace and read r.grounded.score and r.grounded.evidence to see which claims failed

These steps come from our top use case, Flag agent answers that invent facts the tools never returned, which also has a copy-paste agent prompt.

Jevals alternatives · Best AI repos for Software and DevOps

About this repo

  • Who it is forDevelopers and teams building LLM agents or RAG apps who want to eval or guard every trace without paying for a frontier LLM judge on each one.
  • Why it is hot
  • LanguagePython
  • Sourcegithub.com/openlayer-ai/jevals · Back to Hot repos

Requirements

  • Python 3.10 or newer
  • An API key for one backend: AI_GATEWAY_API_KEY (Jev through Vercel AI Gateway), TYPESAFE_API_KEY, or OPENROUTER_API_KEY
  • A trace to evaluate: a messages list in the OpenAI chat format plus the tools you gave the model

Add this badge to your README

Maintain Jevals? Show readers where to find these use cases. Use cases on Deduped

Markdown 

[![Use cases on Deduped](https://dedupedai.com/badge/openlayer-ai/jevals.svg)](https://dedupedai.com/use-cases/openlayer-ai-jevals/?utm_source=github&utm_medium=badge&utm_campaign=openlayer-ai-jevals)

HTML 

<a href="https://dedupedai.com/use-cases/openlayer-ai-jevals/?utm_source=github&amp;utm_medium=badge&amp;utm_campaign=openlayer-ai-jevals"><img src="https://dedupedai.com/badge/openlayer-ai/jevals.svg" alt="Use cases on Deduped" height="20"></a>

A static image, no tracking script. About the badge

Questions about Jevals

What is Jevals?

jevals is a Python library for running agent and quality evals and guardrails by sending all checks for a trace to a decision model in one request. It is an open-source project on GitHub: https://github.com/openlayer-ai/jevals.

Who is Jevals for?

Developers and teams building LLM agents or RAG apps who want to eval or guard every trace without paying for a frontier LLM judge on each one.

What can I do with Jevals?

5 use cases so far, for example: Flag agent answers that invent facts the tools never returned; Catch tool results that try to give the agent orders; Block agent replies that fail checks before they reach the user. Each has steps and a copy-paste agent prompt.

What do I need to run Jevals?

Python 3.10 or newer; An API key for one backend: AI_GATEWAY_API_KEY (Jev through Vercel AI Gateway), TYPESAFE_API_KEY, or OPENROUTER_API_KEY; A trace to evaluate: a messages list in the OpenAI chat format plus the tools you gave the model.

What are alternatives to Jevals?

Other hot AI repos in Software and DevOps: Omarchy Local AI, Hermes Plan Mode, Agent 2, Agnes Harness, Skills.

Is Jevals safe to run?

Deduped does not own, audit or vouch for it. Read the README and license, run it on your own systems first and review every command before you run it.