DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to run 204,800-token context on a 12 GB card with Llam Ampere

Serve a 2.3 bpw Qwen3.8 model with SJ-KVaRN cache on an RTX 3060 12 GB or similar card.

Authorization notice. Llam Ampere is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forA developer with a 12 GB NVIDIA card (RTX 3060 12 GB, 3080 12 GB or 3080 Ti) who wants long-context coding, agentic or RAG work locally.

The problemLong context on a 12 GB card runs out of memory quickly, so most setups cap the context far below what agent or RAG jobs need.

The outcomeA llama-server process with a 204,800-token context that the README reports at about 11 GB of card memory, with the MTP drafter enabled.

What you need first

  • Linux machine with an NVIDIA card (RTX 3090 or 3090 Ti with 24 GB for the main path, or a 12 GB card for the low-memory path)
  • CUDA toolkit (the README tested with 12.4), CMake, git and a C++ compiler
  • Disk space and network access for the model download: 15.6 GB for the 24 GB path, 9.0 GB for the 12 GB path
  • Recommended: OpenSSL development files are not needed if you use the -DLLAMA_BUILD_BORINGSSL=ON build flag, which lets -hf download the model

Steps

  1. Run git clone https://github.com/JakeATX/llamAmpere.git and then cd llamAmpere.
  2. Run cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON. (inferred, check the README)
  3. Run cmake --build build-sm86 -j8 --target llama-server.
  4. Run LLAMA_MTP_DRAFT_COMPUTE_LEAN=1 ./build-sm86/bin/llama-server -hf jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-2.3bpw-MTP-GGUF -hff ATX-Swift-1.5-Qwen3.8-27B-Uncensored-2.3bpw-MTP.gguf -c 204800 --parallel 1 -ngl 99 -fa on -fit off -b 4096 -ub 512 --no-context-shift --jinja -ctk sjkvarn3 -ctv sjkvarn3 --sjkvarn-body-type sjkvarn4t --sjkvarn-sink 128 --sjkvarn-sink-type f16 --sjkvarn-staging-type tq6_0 --sjkvarn-tail 4096 --sjkvarn-tail-max 8192 --cache-type-s q8_0 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0 --spec-draft-vocab-map docs/mtp-vocab/atx_65536.txt --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --spec-draft-window 8192 --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0. (inferred, check the README)
  5. Watch GPU memory with your normal system monitor while a long prompt runs, and compare it with the README figure of about 12,052 MiB for the whole card.

Prompt for your agent

In the llamAmpere repo at <repo path>, build llama-server and start the 2.3 bpw 12 GB command from the README on a 12 GB card. Send a prompt of about <token count> tokens to the local server and report the reply plus the peak GPU memory from nvidia-smi. Verify that the context setting is 204800.

Caveat. The README measured the 12 GB setup on its own cards, and the command has many tuned flags, so a different driver or card may need the README's recommended settings doc instead.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page