# AI use cases for Time to First Token (patchy631/time-to-first-token) Repo: [patchy631/time-to-first-token](https://github.com/patchy631/time-to-first-token) | Use cases: 4 | Page: https://dedupedai.com/use-cases/patchy631-time-to-first-token/ A 10-week self-study roadmap for running LLM inference serving in production, built around one OpenAI-compatible service that gets deployed, instrumented, load tested, optimized, and benchmarked. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Serve a 7-8B model behind an OpenAI-compatible endpoint: https://dedupedai.com/use-cases/patchy631-time-to-first-token/serve-openai-compatible-llm-on-rented-gpu/ - Chart TTFT and queue depth on a live LLM service: https://dedupedai.com/use-cases/patchy631-time-to-first-token/chart-ttft-and-queue-depth-grafana/ - Compare vLLM and SGLang on a shared-prefix workload: https://dedupedai.com/use-cases/patchy631-time-to-first-token/compare-vllm-sglang-shared-prefix-workload/ - Load test an LLM endpoint past 1000 concurrent requests: https://dedupedai.com/use-cases/patchy631-time-to-first-token/load-test-llm-endpoint-past-1000-concurrent/ ## How to serve a 7-8B model behind an OpenAI-compatible endpoint with Time to First Token Repo: [patchy631/time-to-first-token](https://github.com/patchy631/time-to-first-token) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README Deploy vLLM on one rented 24GB GPU and call it with OpenAI-style requests. **Who it is for:** Backend engineer who needs a self-hosted LLM endpoint and has no prior serving experience. **The problem:** Running an LLM in production starts with GPU setup, serving, and API compatibility, and those steps block any real work. **The outcome:** A working endpoint serving Llama-3.1-8B-Instruct or Qwen2.5-7B that existing OpenAI-style client code can call. ### Steps 1. Rent one 24GB GPU by the half hour, for example on RunPod or Modal. 2. `pip install vllm` 3. Start the server with `vllm serve meta-llama/Llama-3.1-8B-Instruct`, using vLLM's OpenAI-compatible server as the README describes. (inferred, check the README) 4. Send one chat request to the endpoint from an OpenAI-compatible client and confirm a response comes back. 5. Save the launch command, since the roadmap reuses it for the following ten weeks. ### Prompt to paste into your agent ```text Using the patchy631/time-to-first-token repo as the guide, set up vLLM on serving Qwen2.5-7B behind an OpenAI-compatible API. Expected output: a chat completion response from the endpoint. Verify that the response contains a non-empty assistant message. ``` ### Caveat The README does not give the exact launch command, so confirm the vLLM flags against the version you install. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/patchy631-time-to-first-token/serve-openai-compatible-llm-on-rented-gpu/ ## How to chart TTFT and queue depth on a live LLM service with Time to First Token Repo: [patchy631/time-to-first-token](https://github.com/patchy631/time-to-first-token) | Difficulty: Intermediate | Time: 1 hour 30 minutes | Basis: From the README Connect Prometheus and Grafana to a vLLM server to watch latency and queueing. **Who it is for:** Platform or SRE engineer who must explain why a self-hosted model feels slow under load. **The problem:** Averages hide the latency users feel, so the team needs time to first token and queue depth on a dashboard. **The outcome:** A Grafana dashboard showing TTFT, inter-token latency, throughput, and queue depth for your own service. ### Steps 1. Keep the vLLM server from the first use case running. 2. Stand up the Prometheus and Grafana stack from the vLLM docs linked in week 3, which requires Docker. 3. Import the dashboard JSON and confirm the panels move under a light trickle of requests. 4. Delete any panel whose metric you cannot define, as the README advises. ### Prompt to paste into your agent ```text In the patchy631/time-to-first-token repo, set up the Prometheus and Grafana stack against a running vLLM server at . Expected output: a Grafana dashboard with live TTFT and queue depth panels. Verify that the TTFT panel and the waiting-requests metric change after you send 20 test requests. ``` ### Caveat The dashboard setup follows the linked vLLM docs, so metric names may change between vLLM releases. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/patchy631-time-to-first-token/chart-ttft-and-queue-depth-grafana/ ## How to compare vLLM and SGLang on a shared-prefix workload with Time to First Token Repo: [patchy631/time-to-first-token](https://github.com/patchy631/time-to-first-token) | Difficulty: Advanced | Time: 4 hours | Basis: From the README Run one model on both engines to measure the TTFT gap on prefix-heavy traffic. **Who it is for:** ML engineer choosing a serving engine who needs evidence from a traffic pattern like their own. **The problem:** Engine performance claims depend on the workload, and the README points to prefix reuse as where RadixAttention helps. **The outcome:** A side-by-side TTFT and throughput comparison of vLLM and SGLang with pinned versions. ### Steps 1. `pip install sglang` 2. Deploy SGLang with the same model, prompt, and hardware used for the vLLM service, so the engine is the only variable. 3. Run a workload with a shared system prompt against both engines. 4. Record TTFT and throughput for each engine and note the versions used. ### Prompt to paste into your agent ```text Using the patchy631/time-to-first-token repo, run the same model with a shared system prompt on vLLM and on SGLang at . Expected output: a table of TTFT and throughput for each engine. Verify that both runs used the same model, prompt set, and hardware before comparing numbers. ``` ### Caveat The README gives no benchmark numbers for this comparison, so results depend on your own model and hardware. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/patchy631-time-to-first-token/compare-vllm-sglang-shared-prefix-workload/ ## How to load test an LLM endpoint past 1000 concurrent requests with Time to First Token Repo: [patchy631/time-to-first-token](https://github.com/patchy631/time-to-first-token) | Difficulty: Advanced | Time: 3 hours | Basis: From the README Use guidellm to sweep concurrency and report p50, p95, and p99 latency. **Who it is for:** Performance engineer preparing an LLM service for peak traffic. **The problem:** Single-request tests miss saturation, so a service can look healthy and still fail under real concurrency. **The outcome:** A reproducible concurrency sweep with p50, p95, and p99 output, cross-checked by a second tool. ### Steps 1. `pip install guidellm` 2. Run a guidellm concurrency sweep against your endpoint, stepping up to 1000 or more concurrent requests. (inferred, check the README) 3. Record p50, p95, and p99 latency at each level, and discard the saturation point from the summary as the README's benchmarking reading recommends. 4. Run a second load tool against the same endpoint and compare the results. ### Prompt to paste into your agent ```text Using the patchy631/time-to-first-token repo, run a guidellm concurrency sweep against the endpoint at up to 1000 concurrent requests. Expected output: p50, p95, and p99 latency at each concurrency level. Verify that the results from a second load tool agree with guidellm within a reasonable margin. ``` ### Caveat The README uses one H100 for this session, so budget for that GPU cost on top of the smaller rental used earlier. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/patchy631-time-to-first-token/load-test-llm-endpoint-past-1000-concurrent/