DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to catch quality regressions before a prompt or model change ships with AI Engineer Notebooks

Run an eval suite on every prompt or model change and compare results to the baseline.

Authorization notice. AI Engineer Notebooks is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forAn AI engineer or tech lead who keeps changing prompts or swapping models and cannot tell whether quality went up or down.

The problemPrompt and model edits fix one case and quietly break others, and nobody measures the change.

The outcomeA repeatable eval run that compares a candidate prompt or model to the current one and flags regressions.

What you need first

  • A Google account to open Colab notebooks
  • A free Groq API key from console.groq.com (no credit card needed)
  • The key saved as a Colab secret, as the setup notebook describes
  • Basic Python and comfort reading API responses

Steps

  1. Open 02-evals-basics/01-measuring-outputs.ipynb and build metrics on a small task before you change anything.
  2. Run 04-evals/02-llm-as-judge.ipynb to set up a judge prompt and check its agreement with human labels.
  3. Open 04-evals/03-regression-evals.ipynb and run the suite on your current prompt to record a baseline.
  4. Change the prompt or model, run the same suite again, and compare the two results.
  5. Block the change if any tracked metric drops below the baseline.

Prompt for your agent

Using the repo calmrocks/ai-engineer-notebooks, run the regression eval in 04-evals/03-regression-evals.ipynb on <placeholder prompt file> before and after a change. Expected output: a table of baseline and candidate scores per metric. Verify that the judge's agreement with human labels was checked before you trust its scores.

Caveat. The regression notebook teaches the pattern of evals as CI, so wiring it into your own pipeline is up to you and the README does not cover that setup.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page