DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to check the retrieval benchmark numbers yourself with PAKGOV RAG Project

Regenerate the results table from committed evaluation files and compare BM25, dense, and hybrid retrieval.

  • Research
  • Intermediate
  • 30 minutes
  • From the README
  • Any agent

Who it is forA researcher or reviewer who needs to verify the R@1, R@5, and MRR figures before citing them in a paper or report.

The problemBenchmark numbers are only useful if they can be checked, and comparing retrievers on different question sets gives misleading rankings.

The outcomeA regenerated FINAL_RESULTS.md table, with the metrics checked against the committed raw outputs in data/interim/ and evaluation/retrieval/.

What you need first

  • Python installed (the README lists Python 3.14)
  • Ability to clone the GitHub repository and run commands in a terminal
  • Internet access for the first download of the sentence-transformers embedding model (inferred from the model being loaded by name)
  • Constitution of Pakistan PDFs or the committed data files for the demo and evaluation

Steps

  1. Clone https://github.com/salikhussain71-code/PAKGOV-RAG-project and open the project folder.
  2. Install the Python packages the evaluation script imports, such as pip install streamlit sentence-transformers faiss-cpu rank_bm25, if the script fails with a missing module. (inferred, check the README)
  3. Run python app/eval/final_table.py.
  4. Open evaluation/retrieval/FINAL_RESULTS.md and compare it with hybrid_rrf_metrics_50.json and dense_metrics_50.json.
  5. Read evaluation/retrieval/failure_analysis_50.md and note the questions where the gold source was not in the top 5.

Prompt for your agent

Clone salikhussain71-code/PAKGOV-RAG-project, run python app/eval/final_table.py, and write the regenerated results to evaluation/retrieval/FINAL_RESULTS.md. Report the R@5 and MRR values for BM25, Dense, and Hybrid RRF (k=60). Verify that the values match the committed metrics JSON files in evaluation/retrieval/ and flag any difference.

Caveat. The README warns that the 50-question set is small, so a 10-point gap at R@5 equals 5 questions and the ranking should be treated as indicative.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page