{
 "schema_version": 1,
 "generated": "2026-10-20",
 "id": "jakeatx-llamampere--run-long-context-model-on-12gb-card",
 "repo": "JakeATX/llamAmpere",
 "repo_url": "https://github.com/JakeATX/llamAmpere",
 "repo_slug": "jakeatx-llamampere",
 "title": "Run 204,800-token context on a 12 GB card",
 "page_title": "How to run 204,800-token context on a 12 GB card with Llam Ampere",
 "summary": "Serve a 2.3 bpw Qwen3.8 model with SJ-KVaRN cache on an RTX 3060 12 GB or similar card.",
 "who": "A developer with a 12 GB NVIDIA card (RTX 3060 12 GB, 3080 12 GB or 3080 Ti) who wants long-context coding, agentic or RAG work locally.",
 "problem": "Long context on a 12 GB card runs out of memory quickly, so most setups cap the context far below what agent or RAG jobs need.",
 "outcome": "A llama-server process with a 204,800-token context that the README reports at about 11 GB of card memory, with the MTP drafter enabled.",
 "steps": [
  "Run `git clone https://github.com/JakeATX/llamAmpere.git` and then `cd llamAmpere`.",
  "Run `cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON`. (inferred, check the README)",
  "Run `cmake --build build-sm86 -j8 --target llama-server`.",
  "Run `LLAMA_MTP_DRAFT_COMPUTE_LEAN=1 ./build-sm86/bin/llama-server -hf jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-2.3bpw-MTP-GGUF -hff ATX-Swift-1.5-Qwen3.8-27B-Uncensored-2.3bpw-MTP.gguf -c 204800 --parallel 1 -ngl 99 -fa on -fit off -b 4096 -ub 512 --no-context-shift --jinja -ctk sjkvarn3 -ctv sjkvarn3 --sjkvarn-body-type sjkvarn4t --sjkvarn-sink 128 --sjkvarn-sink-type f16 --sjkvarn-staging-type tq6_0 --sjkvarn-tail 4096 --sjkvarn-tail-max 8192 --cache-type-s q8_0 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0 --spec-draft-vocab-map docs/mtp-vocab/atx_65536.txt --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --spec-draft-window 8192 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0`. (inferred, check the README)",
  "Watch GPU memory with your normal system monitor while a long prompt runs, and compare it with the README figure of about 12,052 MiB for the whole card."
 ],
 "agent_prompt": "In the llamAmpere repo at <repo path>, build llama-server and start the 2.3 bpw 12 GB command from the README on a 12 GB card. Send a prompt of about <token count> tokens to the local server and report the reply plus the peak GPU memory from nvidia-smi. Verify that the context setting is 204800.",
 "agents": [
  "any"
 ],
 "difficulty": "Advanced",
 "time": "1 hour",
 "basis": "readme",
 "caveat": "The README measured the 12 GB setup on its own cards, and the command has many tuned flags, so a different driver or card may need the README's recommended settings doc instead.",
 "category": "Software and DevOps",
 "tags": [
  "Software and DevOps"
 ],
 "lang": "c++",
 "requirements": [
  "Linux machine with an NVIDIA card (RTX 3090 or 3090 Ti with 24 GB for the main path, or a 12 GB card for the low-memory path)",
  "CUDA toolkit (the README tested with 12.4), CMake, git and a C++ compiler",
  "Disk space and network access for the model download: 15.6 GB for the 24 GB path, 9.0 GB for the 12 GB path",
  "Recommended: OpenSSL development files are not needed if you use the -DLLAMA_BUILD_BORINGSSL=ON build flag, which lets -hf download the model"
 ],
 "risk": "dual-use",
 "seen_in_the_wild": false,
 "mentioned_publicly": false,
 "risks": [],
 "best": false,
 "score": 0.9,
 "evidence": [],
 "url": "https://dedupedai.com/use-cases/jakeatx-llamampere/run-long-context-model-on-12gb-card/",
 "md": "https://dedupedai.com/use-cases/jakeatx-llamampere/run-long-context-model-on-12gb-card/index.md",
 "disclaimer": "Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked \"Seen in the wild\", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email hello@dedupedai.com and we will look at it promptly.",
 "api": "https://dedupedai.com/api/v1/usecases/jakeatx-llamampere--run-long-context-model-on-12gb-card.json",
 "api_version": "v1"
}