DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to score LLM outputs with a custom pass or fail metric with Ragas

Define an Aspect Critique style metric with DiscreteMetric and let an LLM judge label each response.

Short answer: Install Ragas with pip install ragas and set your OPENAI_API_KEY environment variable; then create an LLM with llm_factory and an AsyncOpenAI client as in the README; Define a DiscreteMetric with a name, allowed_values like accurate and inaccurate, and a prompt. All steps.

cost. Every scored response is an LLM call billed to your provider account.

Credentials. The example reads your OPENAI_API_KEY from the environment; keep it out of committed code.

Who it is forA product engineer who wants a quick accurate or inaccurate check on generated summaries.

The problemManual review of every summary does not scale.

The outcomeA label and a written reason for each response you score.

What you need first

  • Python with pip
  • An LLM provider key for LLM-based metrics, for example OPENAI_API_KEY in the README example
  • uv or another way to run the quickstart CLI is optional

Steps

  1. Install Ragas with pip install ragas and set your OPENAI_API_KEY environment variable
  2. Create an LLM with llm_factory and an AsyncOpenAI client as in the README
  3. Define a DiscreteMetric with a name, allowed_values like accurate and inaccurate, and a prompt
  4. Call metric.ascore with the llm and your response, then print score.value and score.reason

Prompt for your agent

With vibrantlabsai/ragas, write a script that defines a DiscreteMetric named summary_accuracy with allowed values accurate and inaccurate, scores each response in <my file of summaries> with metric.ascore, and writes value and reason to a CSV.

Caveat. LLM judges can be wrong; spot check a sample by hand.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for vibrantlabsai/ragas and "Score LLM outputs with a custom pass or fail metric"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer