DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to measure extraction accuracy with the eval harness with Py Idp

Score a backend on field F1 and schema-valid rate before you trust it in production.

Who it is forA developer comparing LLM backends or prompt changes who needs numbers before choosing one for a document workflow.

The problemJudging extraction quality by reading a few outputs does not show how accuracy changes across fields or model versions.

The outcomeA per-field eval report with field F1 and schema-valid rate for the chosen backend.

What you need first

  • Python with pip available
  • An API key for a hosted LLM backend such as OpenAI, or a local backend such as Ollama (the zero-config path runs on the in-tree mock backend without a key)
  • A sample PDF or text file to test with, such as invoice.pdf

Steps

  1. Run pip install py-idp[eval] to add the datasets and pandas packages that idp eval needs.
  2. Run idp eval against your labeled documents, selecting the backend you want to compare.
  3. Repeat the run with a second backend, such as --backend ollama, and keep the report from each run. (inferred, check the README)

Prompt for your agent

In the py-idp repo, run `idp eval` on the labeled set at <path to eval data> using the <backend name> backend. Output the field F1 and schema-valid rate per field. Verify that the report lists every schema field.

Caveat. The README does not list the exact flags for `idp eval` or the label format, so check the CLI help or docs before preparing your dataset.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page