DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to build test data and feedback loops for an LLM app with Ragas

Use Ragas test data generation to cover more scenarios, then rescore after each change.

Short answer: Install Ragas with pip install ragas; then follow the Test Data Generation section of the Ragas docs to create a dataset from your own documents; Score your app against it with the pre-built metrics. All steps.

data. Ragas collects anonymized usage data by default; set RAGAS_DO_NOT_TRACK to true to opt out.

Who it is forA team that has no labeled test set for its LLM feature.

The problemWithout a test set, every prompt change is a guess.

The outcomeA repeatable evaluation run you can compare before and after changes.

What you need first

  • Python with pip
  • An LLM provider key for LLM-based metrics, for example OPENAI_API_KEY in the README example
  • uv or another way to run the quickstart CLI is optional

Steps

  1. Install Ragas with pip install ragas
  2. Follow the Test Data Generation section of the Ragas docs to create a dataset from your own documents
  3. Score your app against it with the pre-built metrics
  4. Rerun after each prompt or retrieval change and compare scores

Prompt for your agent

Using vibrantlabsai/ragas, generate a test dataset from <my documents folder> with its test data generation feature, score my app with the pre-built metrics, and save a baseline report I can compare against next week.

Caveat. The README describes test data generation but its code lives in the linked docs.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for vibrantlabsai/ragas and "Build test data and feedback loops for an LLM app"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer