# AI use cases for Airllm (lyogavin/airllm) Repo: [lyogavin/airllm](https://github.com/lyogavin/airllm) | Use cases: 3 | Page: https://dedupedai.com/use-cases/lyogavin-airllm/ AirLLM cuts inference memory so very large language models run on a single small GPU without quantization, distillation or pruning, by streaming model layers instead of loading the whole model. The README reports 70B models on a 4GB card, DeepSeek-V3 (671B) on about 12GB and Kimi K3 on under 4GB, all through one AutoModel API, plus optional 4-bit or 8-bit compression for speed and new training support for huge models on small VRAM. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Run a 70B-class model on a single 4GB GPU: https://dedupedai.com/use-cases/lyogavin-airllm/run-a-70b-model-on-a-4gb-gpu/ - Speed up large-model inference with 4-bit block compression: https://dedupedai.com/use-cases/lyogavin-airllm/speed-up-airllm-with-compression/ - Run large Llama-style models on a Mac with AirLLM: https://dedupedai.com/use-cases/lyogavin-airllm/run-llama-style-models-on-a-mac/ ## How to run a 70B-class model on a single 4GB GPU with Airllm Repo: [lyogavin/airllm](https://github.com/lyogavin/airllm) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Install AirLLM, pass a Hugging Face repo ID to AutoModel and generate text on a small card. **Who it is for:** A developer with a laptop GPU who wants to try large models locally. **The problem:** Large models normally need far more VRAM than consumer GPUs have, so people rent cloud GPUs to test them. **The outcome:** Text generated by a large model on your own small GPU, with layers streamed from disk. ### Steps 1. Run `pip install airllm` 2. In Python, `from airllm import AutoModel` and load a model with `AutoModel.from_pretrained("Qwen/Qwen3-32B")` or another repo ID 3. Tokenize your prompt with model.tokenizer as in the Quickstart 4. Call model.generate with max_new_tokens and decode the output 5. Set layer_shards_saving_path if you want the split model saved on a different disk ### Prompt to paste into your agent ```text Install lyogavin/airllm and write a script that loads with AutoModel, answers , and prints tokens per second. Warn me first if the model's split shards need more free disk than I have. ``` ### Caveat Layer streaming trades speed for memory, so generation is slow; first runs also split and save the model, which needs a lot of disk. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/lyogavin-airllm/run-a-70b-model-on-a-4gb-gpu/ ## How to speed up large-model inference with 4-bit block compression with Airllm Repo: [lyogavin/airllm](https://github.com/lyogavin/airllm) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Load a model with compression set to 4bit or 8bit for faster runs on the same small GPU. **Who it is for:** A researcher whose AirLLM runs work but are too slow for iteration. **The problem:** Streaming full-precision layers from disk is the bottleneck on small GPUs. **The outcome:** Faster inference with block-wise 4-bit or 8-bit compression on the same hardware. ### Steps 1. Install AirLLM and update bitsandbytes with `pip install -U bitsandbytes` 2. Load your model with compression='4bit' (or '8bit') as in the Model Compression example 3. Run the same prompt with and without compression 4. Compare speed and output quality 5. Keep the setting that gives acceptable quality ### Prompt to paste into your agent ```text Using lyogavin/airllm, load twice, once normally and once with compression='4bit', run the same three prompts on both, and give me a table of time per answer and any quality differences. ``` ### Caveat Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/lyogavin-airllm/speed-up-airllm-with-compression/ ## How to run large Llama-style models on a Mac with AirLLM with Airllm Repo: [lyogavin/airllm](https://github.com/lyogavin/airllm) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Install mlx and torch and run AirLLM on Apple Silicon for supported model architectures. **Who it is for:** A Mac user who wants to try large open models without an NVIDIA GPU. **The problem:** Most large-model tooling assumes CUDA and does not run on Macs. **The outcome:** A Llama-style model running on your Mac through AirLLM's macOS support. ### Steps 1. Install mlx and torch on your Mac 2. Run `pip install airllm` 3. Pick a Llama-style model; only LlamaForCausalLM and MistralForCausalLM run on macOS for now 4. Follow the run_on_macos example notebook from the README 5. Generate text and check memory use while it runs ### Prompt to paste into your agent ```text Set up lyogavin/airllm on my Mac with mlx and torch, choose a Llama-style model that is supported on macOS, run the macOS example with , and report memory use and time per token. ``` ### Caveat Other architectures raise an error on macOS before anything is downloaded. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/lyogavin-airllm/run-llama-style-models-on-a-mac/