# AI use cases for Tracely AI (jwuthri-cloud/Tracely-ai) Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Use cases: 6 | Page: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/ Tracely is a self-hostable tool that grades AI agent traces, clusters failures, freezes bad runs into replayable regression tests, and blocks pull requests that would reintroduce them. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Turn a failing agent trace into a PR-blocking test: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/freeze-failing-trace-into-pr-gate/ - Cluster broken agent runs into one issue per failure: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/group-agent-failures-into-issues/ - Get a Slack or email alert when a live agent run fails: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/alert-on-live-agent-judge-failures/ - Check a judge's accuracy before it blocks a release: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/calibrate-judges-before-they-gate-releases/ - Deploy the full Tracely stack to Railway in one click: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/self-host-tracely-on-railway/ - Let a coding agent query and edit your Tracely workspace: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/let-coding-agent-drive-tracely-via-mcp/ ## How to turn a failing agent trace into a PR-blocking test with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Promote a production failure to a replayable case that blocks the PR that would ship it again. **Who it is for:** An engineer on a team running an AI agent in production who has just seen a real run fail and wants that exact failure to never come back. **The problem:** Hand-writing eval datasets is a guess about what might break, and dashboard numbers move without stopping a release. **The outcome:** The failing run is frozen into a hermetic case with recorded fixtures, and CI fails any pull request that reproduces it. ### Steps 1. Run `docker compose --profile demo up -d --build --wait` in the cloned Tracely repository. 2. Open http://localhost:3001 and find a failing run in the traces view. 3. Promote the failing trace to a regression case in one click. 4. Run `pip install "tracelyai-sdk[openai]"` in your agent project and send traces to the Tracely ingest endpoint. 5. Add the Tracely GitHub Action so `tracely gate` runs on every pull request. ### Prompt to paste into your agent ```text In the Tracely-ai repository, start the demo stack with `docker compose --profile demo up -d --build --wait`, then promote one failing trace in the UI at http://localhost:3001 to a regression case. Report the case name and confirm the case shows a fail-to-pass contract. Verify that the case appears in the cases list before finishing. ``` ### Caveat The README says the case replays against recorded fixtures with no model spend, but you still need your own agent wired to send traces before any failure appears. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/freeze-failing-trace-into-pr-gate/ ## How to cluster broken agent runs into one issue per failure with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Beginner | Time: 30 minutes | Basis: From the README Collapse dozens of failed runs into a single counted issue with a suggested evaluator. **Who it is for:** A team lead or on-call engineer facing a flood of failed agent runs who needs to know which problem is actually happening. **The problem:** Reading failed runs one by one does not show which failure modes are recurring or which check would have caught them. **The outcome:** Failures are grouped by structure and meaning into clusters with counts, each with a suggested evaluator and an option to promote it to a case. ### Steps 1. Run `docker compose --profile demo up -d --build --wait` to load the seeded traces and clusters. 2. Open http://localhost:3001 and go to the clusters view. 3. Compare cluster counts and open the largest cluster to read its trace examples. 4. Check the suggested evaluator for that cluster and add it to the trace table if it fits. 5. Promote the most important cluster to a regression case. ### Prompt to paste into your agent ```text In the Tracely-ai repository, run `docker compose --profile demo up -d --build --wait`, then open the clusters view at http://localhost:3001. List the top three clusters with their counts and suggested evaluators. Verify that the counts add up to the total number of failed traces shown. ``` ### Caveat The demo data is seeded for exploration, so cluster names and counts reflect that sample, not your production agent. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/group-agent-failures-into-issues/ ## How to get a Slack or email alert when a live agent run fails with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Intermediate | Time: 45 minutes | Basis: From the README Build an alert flow that notifies you when a judge fails a live conversation. **Who it is for:** An engineer on call for a production agent who needs to hear about a failing conversation without watching a dashboard. **The problem:** Failures in live agent conversations go unnoticed until a user complains, because nobody is watching the dashboard. **The outcome:** An alert rule fires on a judge failure, runs a flow of conditions and notifications, and sends a message to Slack, email, or your own webhook. ### Steps 1. Run `docker compose --profile demo up -d --build --wait` and open http://localhost:3001. 2. Create an alert rule with a trigger for a judge failure on a live conversation. 3. Add a condition step and a Slack step, filling fields with template chips from the failure variables. 4. Use the test run to see exactly what each step sent. 5. Save the rule and confirm it is active. ### Prompt to paste into your agent ```text In the Tracely-ai repository, start the stack with `docker compose --profile demo up -d --build --wait` and create an alert rule at http://localhost:3001 that fires when an evaluator fails a live conversation and sends a Slack message. Report the rule name and the output of a test run. Verify that the test run shows the payload each step sent. ``` ### Caveat The README describes the alert trigger and flow options, but a live Slack or email delivery also needs your own accounts and credentials configured outside this repo. Risk, credentials: Slack webhooks and custom webhook headers can carry secrets, so store them only in the alert configuration and never commit them to a repository. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/alert-on-live-agent-judge-failures/ ## How to check a judge's accuracy before it blocks a release with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Label judge verdicts against human review to catch over-flagging before it gates a PR. **Who it is for:** A QA or ML engineer who has added an LLM judge to a release gate and wants proof it flags the right runs. **The problem:** An LLM judge that over-flags will block good pull requests, and an under-flagging judge lets regressions ship. **The outcome:** Each judge gets an agreement score against human labels, with missed failures and over-flagging shown separately. ### Steps 1. Run `docker compose --profile demo up -d --build --wait` and open http://localhost:3001. 2. Open the calibration screen. 3. Label a set of judge verdicts as pass or fail based on your own review. 4. Read the per-evaluator agreement, missed failures and over-flagging counts. 5. Only attach a judge to the CI gate if its agreement is acceptable to your team. ### Prompt to paste into your agent ```text In the Tracely-ai repository, run `docker compose --profile demo up -d --build --wait` and open the calibration screen at http://localhost:3001. Label at least ten judge verdicts and report each evaluator's agreement, missed failures and over-flagging. Verify that the three numbers are shown for every evaluator you labeled. ``` ### Caveat Calibration is only as good as the human labels you provide, so the agreement numbers reflect your reviewers' judgment as much as the judge. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/calibrate-judges-before-they-gate-releases/ ## How to deploy the full Tracely stack to Railway in one click with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Run the API, worker, UI, Postgres, ClickHouse, Redis and MinIO on your own Railway project. **Who it is for:** A platform or DevOps engineer who needs the agent observability stack hosted in their own cloud account. **The problem:** Teams that cannot send agent traces to a hosted service need the whole stack running inside their own infrastructure. **The outcome:** A running Tracely deployment on Railway with a frontend domain where you can create your workspace. ### Steps 1. Click the Railway deploy button in the Tracely README. 2. Generate the two secrets with `openssl rand -hex 32` and set them as SESSION_SECRET and SECRETS_ENCRYPTION_KEY when prompted. 3. Wait for the first deploy to finish, which runs migrations and seeding. 4. Open the frontend domain and create your workspace. 5. Create an ingest key and point your agent's OTLP exporter at the backend. ### Prompt to paste into your agent ```text For the Tracely-ai repository, use the Railway deploy template linked in the README, generate SESSION_SECRET and SECRETS_ENCRYPTION_KEY with `openssl rand -hex 32`, and report the frontend domain once the deploy succeeds. Verify that the frontend loads and that you can create a workspace before finishing. ``` ### Caveat The README says the deploy runs migrations and seeding automatically, but you must set the two secrets yourself before the first deploy succeeds. Risk, credentials: The two secrets protect sessions and stored credentials, so keep the values in your Railway variables and never paste them into chat or commit them. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/self-host-tracely-on-railway/ ## How to let a coding agent query and edit your Tracely workspace with Tracely AI Repo: [jwuthri-cloud/Tracely-ai](https://github.com/jwuthri-cloud/Tracely-ai) | Difficulty: Advanced | Time: 45 minutes | Basis: From the README Connect a coding agent to the Tracely MCP endpoint with an ingest key. **Who it is for:** A developer who uses a coding agent day to day and wants it to inspect failing traces and set up cases without leaving the editor. **The problem:** Switching between the editor and the Tracely UI slows down debugging of agent failures. **The outcome:** A coding agent reads traces, creates evaluators, cases and alerts in your workspace through the MCP endpoint. ### Steps 1. Run `docker compose --profile demo up -d --build --wait` to start a local Tracely instance. 2. Note the ingest key from the seeded project, which the README lists as `tracely_dev_key` for local use. 3. Add the MCP endpoint at http://localhost:8000/mcp to your coding agent's MCP settings with that key. 4. Ask the agent to list recent failing traces and summarize them. 5. Review anything the agent creates in the Tracely UI before relying on it. ### Prompt to paste into your agent ```text Using the Tracely-ai repository running locally at http://localhost:8000, connect to the MCP endpoint at /mcp with the ingest key . List the five most recent failing traces and report their agent names and failure reasons. Verify that the list matches what the traces view shows at http://localhost:3001. ``` ### Caveat The README says the MCP endpoint uses an ordinary ingest key, but the exact client configuration for each coding agent is not in the README and must be set up by you. Risk, credentials: The ingest key gives the connected agent access to your workspace, so use a key scoped for development and rotate it if it leaks. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/jwuthri-cloud-tracely-ai/let-coding-agent-drive-tracely-via-mcp/