DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to score your app's answers with evals and datasets with Langfuse

Build a dataset of test cases and score outputs with LLM-as-a-judge, code evaluators and user feedback.

Short answer: Run Langfuse and trace your app; then create a Dataset of representative inputs and expected outputs; run experiments over the dataset for each prompt or model version. All steps.

cost. LLM-as-a-judge evaluators call a model for every scored item, which costs money with a paid provider.

Who it is forA team about to swap models who needs proof the new one is not worse.

The problemQuality is judged by gut feel, so regressions slip into production.

The outcomeDatasets and evaluation scores you can compare across prompt and model versions before deploying.

What you need first

  • Docker and Docker Compose for local self-hosting, or a Langfuse Cloud account
  • The Python or JS/TS SDK, or a supported framework integration
  • For production: a VM, Kubernetes with Helm, or Terraform on AWS, Azure or GCP

Steps

  1. Run Langfuse and trace your app
  2. Create a Dataset of representative inputs and expected outputs
  3. Run experiments over the dataset for each prompt or model version
  4. Score results with LLM-as-a-judge, code evaluators, manual labeling or user feedback collection
  5. Use the API to wire these runs into your own LLMOps workflow

Prompt for your agent

Using langfuse/langfuse, create a dataset from <examples>, run it against <current model> and <new model>, and score both with an LLM-as-a-judge evaluator for <criteria>. Summarise which version wins and on which cases.

Caveat. A judge model has its own biases, so spot-check its scores by hand.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for langfuse/langfuse and "Score your app's answers with evals and datasets"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer