{
 "schema_version": 1,
 "generated": "2026-10-31",
 "id": "zinyy-adaflash--benchmark-adaflash-speedup-on-gsm8k",
 "repo": "ZinYY/AdaFlash",
 "repo_url": "https://github.com/ZinYY/AdaFlash",
 "repo_slug": "zinyy-adaflash",
 "title": "Measure Ada Flash decoding speed against baselines on GSM8K",
 "page_title": "How to measure Ada Flash decoding speed against baselines on GSM8K with Ada Flash",
 "summary": "Run the one-command Ada Flash benchmark on a GSM8K split to get throughput numbers for your own GPUs.",
 "who": "An ML engineer deciding whether Ada Flash is worth adopting for a Qwen3-8B serving setup and needing numbers from their own hardware.",
 "problem": "Published speedups may not match your hardware, batch sizes, or workload, so you need a repeatable local measurement before changing serving infrastructure.",
 "outcome": "A benchmark run over 1024 GSM8K samples at concurrency 64 with the Ada Flash drafter, plus a documented protocol for comparing against other methods.",
 "steps": [
  "Create the environment with `conda create -n adaflash python=3.12 -y` and then `conda activate adaflash`.",
  "Install the patched SGLang with `pip install \"sglang[all] @ git+https://github.com/sgl-project/sglang.git@refs/pull/34171/head#subdirectory=python\"` and then `pip install -r requirements.txt`.",
  "Prepare the benchmark split with `python bin/prepare_data.py --dataset gsm8k`.",
  "Set `export MODEL_PATH=Qwen/Qwen3-8B`, `export DRAFT_MODEL_PATH=AdaFlash/Qwen3-8B-PerfectBlend`, `export DATASET=gsm8k`, `export NUM_SAMPLES=1024`, and `export CONCURRENCY=64`.",
  "Run `bash scripts/tools/benchmark.sh` and read the results it writes, then follow docs/benchmark_experiments.md to run the other baselines under the same settings."
 ],
 "agent_prompt": "In the ZinYY/AdaFlash repo, set up the environment from the README, prepare the gsm8k split, and run scripts/tools/benchmark.sh with MODEL_PATH=Qwen/Qwen3-8B and DRAFT_MODEL_PATH=AdaFlash/Qwen3-8B-PerfectBlend for <your GPU machine>. Report the throughput output from the benchmark. Verify that the server started with the patched SGLang before trusting any number.",
 "agents": [
  "any"
 ],
 "difficulty": "Intermediate",
 "time": "1 hour",
 "basis": "readme",
 "caveat": "The README says a stock SGLang release may not work, so the patched SGLang PR must be installed before the benchmark will run.",
 "category": "Software and DevOps",
 "tags": [
  "Software and DevOps"
 ],
 "lang": "python",
 "requirements": [
  "A Linux machine with NVIDIA GPUs and CUDA, since the README sets CUDA devices and separate GPU sets for serving and training",
  "Conda and Python 3.12, as used in the README install steps",
  "Access to the Qwen3-8B target model and the AdaFlash/DFlash checkpoints on Hugging Face",
  "The patched SGLang version from the linked SGLang PR, because a stock SGLang release may not expose the required arguments"
 ],
 "risk": "none",
 "seen_in_the_wild": false,
 "mentioned_publicly": false,
 "risks": [],
 "best": true,
 "score": 1.0,
 "evidence": [],
 "url": "https://dedupedai.com/use-cases/zinyy-adaflash/benchmark-adaflash-speedup-on-gsm8k/",
 "md": "https://dedupedai.com/use-cases/zinyy-adaflash/benchmark-adaflash-speedup-on-gsm8k/index.md",
 "disclaimer": "Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked \"Seen in the wild\", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email hello@dedupedai.com and we will look at it promptly.",
 "api": "https://dedupedai.com/api/v1/usecases/zinyy-adaflash--benchmark-adaflash-speedup-on-gsm8k.json",
 "api_version": "v1"
}