# AI use cases for AI Engineer Notebooks (calmrocks/ai-engineer-notebooks) Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Use cases: 5 | Page: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/ A set of free, framework-free Jupyter notebooks in Google Colab that teach the applied-LLM stack for AI Engineer and Forward Deployed Engineer roles, running on the free Groq API. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Build a RAG question-answering prototype with a golden set: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/build-rag-qa-with-golden-set/ - Catch quality regressions before a prompt or model change ships: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/gate-prompt-changes-with-regression-evals/ - Test an agent against prompt injection before it touches real data: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/test-agent-against-prompt-injection/ - Decide between a fixed pipeline and an agent using cost and latency data: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/decide-pipeline-or-agent-with-cost-data/ - Get reliable JSON and tool calls from a model in a backend service: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/add-structured-output-and-tool-calls/ ## How to build a RAG question-answering prototype with a golden set with AI Engineer Notebooks Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Build retrieval-augmented answers from raw API calls, then score them against a golden set. **Who it is for:** A backend engineer assigned a first document Q&A feature who needs to show it works before it ships. **The problem:** A plain model call invents answers about your documents, and a demo that looks right often fails on real questions. **The outcome:** A working retrieve, augment, generate loop with a golden set of questions and a measured pass rate. ### Steps 1. Open the setup notebook at 00-setup/00-environment.ipynb from the Colab badge and add your Groq key as a Colab secret. 2. Run 03-rag/00-what-is-rag.ipynb to see the retrieve, augment, generate loop in about 15 lines. 3. Work through 03-rag/01-embeddings-retrieval.ipynb until retrieval returns the right passages for your questions. 4. Open 04-evals/01-golden-sets.ipynb and build a golden set of questions with expected answers for your RAG system. 5. Run 03-rag/04-why-rag-fails.ipynb on any bad answers to find out whether retrieval or generation is the bottleneck. ### Prompt to paste into your agent ```text Using the repo calmrocks/ai-engineer-notebooks, work through 03-rag/00-what-is-rag.ipynb and 04-evals/01-golden-sets.ipynb against the documents in . Report the golden-set pass rate and list the three questions that failed. Verify that each failed answer is checked against the retrieved passages, not only the final text. ``` ### Caveat The notebooks use the Groq API on Colab, so you will need your own free Groq key and a Google account before the first run. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/build-rag-qa-with-golden-set/ ## How to catch quality regressions before a prompt or model change ships with AI Engineer Notebooks Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Difficulty: Advanced | Time: 1 hour | Basis: From the README Run an eval suite on every prompt or model change and compare results to the baseline. **Who it is for:** An AI engineer or tech lead who keeps changing prompts or swapping models and cannot tell whether quality went up or down. **The problem:** Prompt and model edits fix one case and quietly break others, and nobody measures the change. **The outcome:** A repeatable eval run that compares a candidate prompt or model to the current one and flags regressions. ### Steps 1. Open 02-evals-basics/01-measuring-outputs.ipynb and build metrics on a small task before you change anything. 2. Run 04-evals/02-llm-as-judge.ipynb to set up a judge prompt and check its agreement with human labels. 3. Open 04-evals/03-regression-evals.ipynb and run the suite on your current prompt to record a baseline. 4. Change the prompt or model, run the same suite again, and compare the two results. 5. Block the change if any tracked metric drops below the baseline. ### Prompt to paste into your agent ```text Using the repo calmrocks/ai-engineer-notebooks, run the regression eval in 04-evals/03-regression-evals.ipynb on before and after a change. Expected output: a table of baseline and candidate scores per metric. Verify that the judge's agreement with human labels was checked before you trust its scores. ``` ### Caveat The regression notebook teaches the pattern of evals as CI, so wiring it into your own pipeline is up to you and the README does not cover that setup. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/gate-prompt-changes-with-regression-evals/ ## How to test an agent against prompt injection before it touches real data with AI Engineer Notebooks Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Difficulty: Advanced | Time: 45 minutes | Basis: From the README Run direct and indirect injection attacks on a tool-using agent, then add defenses and retest. **Who it is for:** A security-minded engineer who has been asked to sign off on an LLM feature that reads external content and calls tools. **The problem:** An agent that reads web pages, emails, or documents can be steered by text hidden in that content into misusing its tools. **The outcome:** A list of working attacks against the agent, the defenses that stopped them, and the attacks that still got through. ### Steps 1. Open 07-security/01-prompt-injection-and-trust.ipynb and run the direct and indirect injection examples against the sample agent. 2. Note which outputs are handled unsafely and which tool calls the attacks trigger. 3. Open 05-agents/03-guardrails-and-budgets.ipynb and add stopping conditions and budgets to limit what the agent can do. 4. Rerun the injection examples and record which attacks still succeed. 5. Write down the trust boundary for each tool and the remaining gaps for your review. ### Prompt to paste into your agent ```text Using the repo calmrocks/ai-engineer-notebooks, run the injection examples in 07-security/01-prompt-injection-and-trust.ipynb against . Expected output: a table of attacks with the tool calls each one triggered, before and after defenses. Verify that the test covers both direct and indirect injection. ``` ### Caveat The security notebook teaches the OWASP LLM Top 10 risks on the notebook's own sample agent, so it does not test your production system until you point it there. Risk, dual-use: The injection examples are working attack payloads, so run them only against systems you own or are authorized to test. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/test-agent-against-prompt-injection/ ## How to decide between a fixed pipeline and an agent using cost and latency data with AI Engineer Notebooks Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Difficulty: Advanced | Time: 1 hour | Basis: From the README Compare a fixed pipeline with an agent loop on cost, latency, and quality before you pick one. **Who it is for:** A solutions engineer or architect who has to recommend an architecture for a customer task and must justify the cost. **The problem:** Teams often build an agent when a fixed pipeline would do, which adds calls, latency, and cost without a quality gain. **The outcome:** A side-by-side comparison of a pipeline and an agent on the same task, with a reasoned recommendation. ### Steps 1. Run 05-agents/01-agent-loop-from-scratch.ipynb to build a working agent loop with raw API calls. 2. Build the same task as a fixed pipeline using the RAG and tool calling patterns from sections 01 and 03. 3. Open 05-agents/03-guardrails-and-budgets.ipynb and read the section on when a pipeline beats an agent. 4. Measure cost, latency, and answer quality for both versions on the same golden set. 5. Recommend the cheaper option that meets the quality bar and record the numbers. ### Prompt to paste into your agent ```text Using the repo calmrocks/ai-engineer-notebooks, build the agent loop from 05-agents/01-agent-loop-from-scratch.ipynb and a fixed pipeline for . Expected output: a table comparing calls per task, latency, and quality for each. Verify that both versions were scored on the same set of inputs. ``` ### Caveat The README names the pipeline-vs-agent cost comparison as a case study but does not give its results, so your numbers will come from your own runs. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/decide-pipeline-or-agent-with-cost-data/ ## How to get reliable JSON and tool calls from a model in a backend service with AI Engineer Notebooks Repo: [calmrocks/ai-engineer-notebooks](https://github.com/calmrocks/ai-engineer-notebooks) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README Make a model return valid structured output and handle tool-call errors in a service. **Who it is for:** A backend engineer adding an LLM step to an existing service that needs typed data back from the model. **The problem:** Model output that is almost JSON breaks parsers, and tool calls fail in ways the service does not handle. **The outcome:** A tested integration that parses structured output, calls tools, and handles the error paths. ### Steps 1. Run 01-model-apis/00-prompting-basics.ipynb to see how output-format specs change the result. 2. Open 01-model-apis/01-structured-output.ipynb and find where the JSON breaks for your task. 3. Work through 01-model-apis/02-tool-calling.ipynb, including the error paths. 4. Add the reliability patterns from 08-operations/02-reliability-and-fallbacks.ipynb, such as retries and output validation. 5. Run the golden set from 04-evals/01-golden-sets.ipynb to confirm the parser handles real inputs. ### Prompt to paste into your agent ```text Using the repo calmrocks/ai-engineer-notebooks, work through 01-model-apis/01-structured-output.ipynb and 01-model-apis/02-tool-calling.ipynb for . Expected output: a parser that returns valid objects for every test input and a log of tool-call errors handled. Verify that malformed model output is caught and retried rather than passed on. ``` ### Caveat The notebooks run on Colab with the Groq API, so moving the code into your own service is a separate step the README does not cover. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/calmrocks-ai-engineer-notebooks/add-structured-output-and-tool-calls/