# AI use cases for llama.cpp (ggml-org/llama.cpp)

Repo: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | Use cases: 3 | Page: https://dedupedai.com/use-cases/ggml-org-llama-cpp/

llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models locally or in the cloud, with quantization, many GPU backends, a CLI and an OpenAI-compatible server.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Run an open model locally in one command: https://dedupedai.com/use-cases/ggml-org-llama-cpp/chat-with-local-model-from-hugging-face/
- Serve a local model behind an OpenAI-compatible API: https://dedupedai.com/use-cases/ggml-org-llama-cpp/serve-openai-compatible-local-api/
- Fit a bigger model on limited hardware: https://dedupedai.com/use-cases/ggml-org-llama-cpp/run-bigger-models-with-quantization/

## How to run a GGUF model from Hugging Face with llama.cpp

Repo: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README

Install llama.cpp and chat with a GGUF model pulled straight from Hugging Face.

**Who it is for:** A developer who wants to try a local model without Python environments.

**The problem:** Setting up local inference often means fighting dependencies.

**The outcome:** A local chat session with a downloaded model.

### Steps

1. Install with `curl -LsSf https://llama.app/install.sh | sh` (or `irm https://llama.app/install.ps1 | iex` on Windows)
2. Run `llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF`
3. Chat in the terminal
4. Try a larger GGUF model if your hardware allows

### Prompt to paste into your agent

```text
Install ggml-org/llama.cpp and run `llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF`. Ask it <question> and report tokens per second shown, confirming the model ran locally.
```

### Caveat

Pre-built binaries and Docker are alternatives if you prefer not to run the install script.

Risk, remote script: The quick install pipes a script from llama.app into your shell, so review it first or use the release binaries.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/ggml-org-llama-cpp/chat-with-local-model-from-hugging-face/

## llama.cpp OpenAI compatible server: serve a local model

Repo: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | Difficulty: Intermediate | Time: 15 minutes | Basis: From the README

Start llama serve to give your apps an OpenAI-style endpoint and a built-in web UI.

**Who it is for:** A developer whose app uses the OpenAI SDK and who wants to test against a free local model.

**The problem:** Paying for API calls during development adds up.

**The outcome:** A local OpenAI-compatible server your existing code can point at.

### Steps

1. Install llama.cpp
2. Run `llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF`
3. Open the built-in web UI to test
4. Point your OpenAI client base URL at the local server

### Prompt to paste into your agent

```text
Start `llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF`, then send one chat completion from <app> to it. Confirm the response came from the local server, not a cloud API.
```

### Caveat

Small models like the 0.8B example are fast but much weaker than hosted frontier models.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/ggml-org-llama-cpp/serve-openai-compatible-local-api/

## llama.cpp quantization: run models bigger than your VRAM

Repo: [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README

Use low-bit quantized GGUF models and CPU plus GPU hybrid inference to run models larger than your VRAM.

**Who it is for:** An ML hobbyist with an 8 GB GPU who wants to try a mid-size model.

**The problem:** Full-precision models do not fit consumer GPUs.

**The outcome:** A larger model running with partial GPU acceleration.

### Steps

1. Install llama.cpp with the backend for your GPU (CUDA, Metal, Vulkan, HIP and more)
2. Choose a 4-bit or lower quantized GGUF of the model
3. Run it with llama cli and let layers spill to CPU
4. Check the performance troubleshooting doc if it is slow

### Prompt to paste into your agent

```text
Run a 4-bit GGUF of <model> with llama.cpp on a machine with <VRAM> GB VRAM using CPU+GPU hybrid inference. Report tokens per second and confirm the model loaded fully.
```

### Caveat

Lower-bit quantization reduces memory use but can reduce output quality.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/ggml-org-llama-cpp/run-bigger-models-with-quantization/