DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to run a 70B-class model on a single 4GB GPU with Airllm

Install AirLLM, pass a Hugging Face repo ID to AutoModel and generate text on a small card.

Short answer: Run pip install airllm; then in Python, from airllm import AutoModel and load a model with AutoModel.from_pretrained("Qwen/Qwen3-32B") or another repo ID; Tokenize your prompt with model.tokenizer as in the Quickstart. All steps.

Who it is forA developer with a laptop GPU who wants to try large models locally.

The problemLarge models normally need far more VRAM than consumer GPUs have, so people rent cloud GPUs to test them.

The outcomeText generated by a large model on your own small GPU, with layers streamed from disk.

What you need first

  • Python and pip
  • A CUDA GPU (a few GB of VRAM) or a Mac with mlx and torch for Llama-style models
  • Lots of disk for split layer shards; some models need specific transformers versions
  • Optional: bitsandbytes for compression

Steps

  1. Run pip install airllm
  2. In Python, from airllm import AutoModel and load a model with AutoModel.from_pretrained("Qwen/Qwen3-32B") or another repo ID
  3. Tokenize your prompt with model.tokenizer as in the Quickstart
  4. Call model.generate with max_new_tokens and decode the output
  5. Set layer_shards_saving_path if you want the split model saved on a different disk

Prompt for your agent

Install lyogavin/airllm and write a script that loads <Hugging Face model ID> with AutoModel, answers <prompt>, and prints tokens per second. Warn me first if the model's split shards need more free disk than I have.

Caveat. Layer streaming trades speed for memory, so generation is slow; first runs also split and save the model, which needs a lot of disk.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for lyogavin/airllm and "Run a 70B-class model on a single 4GB GPU"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer