# AI use cases for Jevals (openlayer-ai/jevals)

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Use cases: 5 | Page: https://dedupedai.com/use-cases/openlayer-ai-jevals/

jevals is a Python library for running agent and quality evals and guardrails by sending all checks for a trace to a decision model in one request.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Flag agent answers that invent facts the tools never returned: https://dedupedai.com/use-cases/openlayer-ai-jevals/flag-unsupported-agent-claims/
- Catch tool results that try to give the agent orders: https://dedupedai.com/use-cases/openlayer-ai-jevals/screen-tool-results-for-injected-instructions/
- Block agent replies that fail checks before they reach the user: https://dedupedai.com/use-cases/openlayer-ai-jevals/gate-agent-loop-before-it-answers/
- Let Cursor or Claude Code write and run agent evals for you: https://dedupedai.com/use-cases/openlayer-ai-jevals/write-and-run-evals-from-your-coding-agent/
- Run agent evals on your own Mac with no API calls: https://dedupedai.com/use-cases/openlayer-ai-jevals/run-evals-fully-local-on-a-mac/

## How to flag agent answers that invent facts the tools never returned with Jevals

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Score each claim in an agent's final answer against its tool results in one request.

**Who it is for:** A developer whose agent answers questions from search or database tools and sometimes states things the tools did not return.

**The problem:** Relevance-style metrics can pass an answer that is on-topic but partly invented, so the made-up claim goes unnoticed.

**The outcome:** A per-claim support score and a list of unsupported claims for each trace, plus a check on whether the answer used the tool output at all.

### Steps

1. `pip install jevals`
2. `export AI_GATEWAY_API_KEY=...` (TYPESAFE_API_KEY or OPENROUTER_API_KEY also work)
3. Save a script that calls `evaluate({"messages": messages, "tools": tools}, [UsedToolResult(), Grounded()])` with `from jevals.agent import UsedToolResult, Grounded` (inferred, check the README)
4. Run the script on one saved trace and read `r.grounded.score` and `r.grounded.evidence` to see which claims failed

### Prompt to paste into your agent

```text
Use the jevals repo to evaluate the agent trace in <trace_file>. Call evaluate() with UsedToolResult() and Grounded() on its messages and tools. Report the grounded score and every claim with a low support probability. Verify the report lists the claims in the order they appear in the final answer.
```

### Caveat

The emulated chat LLM backend returns 0.00 or 1.00 instead of calibrated probabilities, so use a decision model backend for real scores.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/openlayer-ai-jevals/flag-unsupported-agent-claims/

## How to catch tool results that try to give the agent orders with Jevals

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Check every tool result for indirect prompt injection and PHI in the same request as your other evals.

**Who it is for:** A security or platform engineer running an agent that reads web pages, emails, or documents it did not write.

**The problem:** Text returned by a tool can contain instructions meant to hijack the agent, and the agent may follow them without the user noticing.

**The outcome:** A pass or fail for indirect injection on each trace, plus a PHI check, with the probability the decision model returned.

### Steps

1. `pip install jevals`
2. `pip install "jevals[pii]"` to add Presidio for PII and PHI entity detection
3. `export AI_GATEWAY_API_KEY=...`
4. Call `evaluate(sample, [IndirectInjection(), PHI()])` with `from jevals.security import IndirectInjection, PHI` on your saved traces (inferred, check the README)
5. Read `r.indirect_injection.passed` and `r.phi.passed`, and review any failing traces by hand

### Prompt to paste into your agent

```text
In the jevals repo, run IndirectInjection() and PHI() from jevals.security against the traces in <traces_file>. Report each trace that fails indirect_injection, with the tool result text that triggered it. Verify that the PHI check ran with the jevals[pii] extra installed.
```

### Caveat

The README gives a probability for each check, not a guarantee, so treat a pass as a screen and not as proof that a trace is safe.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/openlayer-ai-jevals/screen-tool-results-for-injected-instructions/

## How to block agent replies that fail checks before they reach the user with Jevals

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Difficulty: Advanced | Time: 1 hour | Basis: From the README

Run evals inside an async agent loop with aevaluate() or a gate's acheck().

**Who it is for:** A developer shipping an agent whose replies must pass scope and relevance checks before they go out.

**The problem:** Offline evals catch problems after users see them, and slow judges are too expensive to run on every reply.

**The outcome:** A gate in the request path that checks each reply and can stop or retry it, using the same eval definitions you used offline.

### Steps

1. `pip install jevals`
2. `export AI_GATEWAY_API_KEY=...`
3. Define your checks with `StayedInScope()` and `AnswerRelevancy()` from `jevals.agent` and `jevals.quality`
4. Inside your async agent loop, await `aevaluate()` on the current messages and tools, or call a gate's `acheck()`
5. Branch on the result: return the reply if it passes, otherwise regenerate or fall back

### Prompt to paste into your agent

```text
Wire jevals into the async agent loop in <agent_file>. Use aevaluate() with StayedInScope() and AnswerRelevancy() on each reply before it is returned. Report the latency added per reply. Verify that a reply that fails either check is not sent to the user.
```

### Caveat

Calibrated probabilities depend on the backend, so run Jevals calibrate again if you switch backends before relying on a gate threshold.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/openlayer-ai-jevals/gate-agent-loop-before-it-answers/

## How to let Cursor or Claude Code write and run agent evals for you with Jevals

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README

Install the Jevals MCP server so your coding agent can run and author evals.

**Who it is for:** A developer who works in Cursor or Claude Code and wants evals added to an agent project without writing them by hand.

**The problem:** Writing eval classes and questions from scratch is slow, and the eval code is easy to get wrong.

**The outcome:** Your coding agent creates eval definitions for your agent and runs them on sample traces from inside the editor.

### Steps

1. `pip install "jevals[mcp]"`
2. `export AI_GATEWAY_API_KEY=...`
3. Add the Jevals MCP server to the MCP settings of Cursor or Claude Code, following that client's documentation (inferred, check the README)
4. Ask the coding agent to write evals for your agent's traces and run them with `evaluate()`

### Prompt to paste into your agent

```text
Using the jevals MCP server, write an eval for <agent_repo> that checks whether the agent used its tool results. Run it on one trace from <traces_file>. Report the score and the trace id. Verify the eval runs without a syntax error before you report.
```

### Caveat

The README does not say how to register the MCP server in each client, so check that client's setup before you assume it is connected.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/openlayer-ai-jevals/write-and-run-evals-from-your-coding-agent/

## How to run agent evals on your own Mac with no API calls with Jevals

Repo: [openlayer-ai/jevals](https://github.com/openlayer-ai/jevals) | Difficulty: Advanced | Time: 1 hour | Basis: From the README

Serve Kev or use Laya on Apple Silicon so evals run locally.

**Who it is for:** A developer with a 32GB Mac or Apple Silicon machine who cannot send agent traces to a hosted API.

**The problem:** Sensitive traces may not be allowed to leave the machine, and hosted backends need keys and network access.

**The outcome:** Evals run against a local decision model, with traces kept on your machine.

### Steps

1. `pip install jevals`
2. For Kev: `python -m kev.serve --run jaredpalmer/kev-4b` on a 32GB Mac, then point evaluate() at `backend="kev://localhost:8009"`
3. For Laya: `pip install "jevals[laya]"` on Apple Silicon, then set `JEVALS_BACKEND=laya`
4. Run `evaluate()` on a saved trace and check the result against a few traces you have already reviewed by hand

### Prompt to paste into your agent

```text
Set up jevals to run locally on this Mac using Kev or Laya. Serve the decision model, run evaluate() with ToolChoice() and Grounded() on <trace_file>, and report the local latency. Verify that no request leaves the machine by checking the backend URL is localhost.
```

### Caveat

The README says probabilities differ across decision models, so re-run Jevals calibrate after switching to Kev or Laya.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/openlayer-ai-jevals/run-evals-fully-local-on-a-mac/