DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

How to score which retrieval setup works best on your knowledge base with Deep Tutor

Write a small QA set and run deeptutor kb eval to get Recall, Precision, nDCG and MRR.

Short answer: Write a QA set as JSON lines, each with a query and the gold passages an ideal retrieval should return; then run deeptutor kb eval <name> --dataset qa.jsonl --top-k 5 [--mode hybrid] [--save baseline.json]; Change the engine or mode and save a second report. All steps.

  • Research
  • Advanced
  • 45 minutes
  • From the README
  • Claude Code
  • Codex

Who it is forA teacher or researcher deciding how to index course material in Deep Tutor.

The problemIt is unclear which retrieval engine or mode actually finds the right passages for your documents.

The outcomeSaved metric reports you can compare per metric and per case.

What you need first

  • Docker, or Python 3.11 to 3.14 with Node.js 20.9+ for the PyPI web app
  • An LLM provider and model configured in Settings, Providers
  • Optional embedding and search providers for knowledge bases

Steps

  1. Write a QA set as JSON lines, each with a query and the gold passages an ideal retrieval should return
  2. Run deeptutor kb eval <name> --dataset qa.jsonl --top-k 5 [--mode hybrid] [--save baseline.json]
  3. Change the engine or mode and save a second report
  4. Compare them with deeptutor kb eval-diff baseline.json candidate.json

Prompt for your agent

For my DeepTutor knowledge base <name>, write 10 QA cases from these documents as qa.jsonl, run `deeptutor kb eval` with --top-k 5 and --save baseline.json, and summarize Recall@k and MRR.

Caveat. Matching uses exact text or lexical overlap; semantic equivalence is not assessed, and PageIndex is excluded.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for HKUDS/DeepTutor and "Score which retrieval setup works best on your knowledge base"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer