# AI use cases for Lithos Metal (lithos-ai/lithos-metal)

Repo: [lithos-ai/lithos-metal](https://github.com/lithos-ai/lithos-metal) | Use cases: 4 | Page: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/

lithos-metal is an open-source LLM inference engine that generates Metal megakernels and serves local models on Apple silicon Macs.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Run a local model as the backend for a coding agent: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/local-coding-agent-backend/
- Call the local model from scripts over HTTP: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/call-local-api-from-scripts/
- Compare speculative and target-only generation: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/compare-draft-and-target-only-generation/
- Build Lithos Metal from source for development: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/build-from-source-for-development/

## How to run a local model as the backend for a coding agent with Lithos Metal

Repo: [lithos-ai/lithos-metal](https://github.com/lithos-ai/lithos-metal) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Serve Qwen3.8-27B on your Mac and point a coding agent at it without sending code to a hosted API.

**Who it is for:** A developer on an Apple silicon Mac who wants a coding agent to run against a model that stays on the local machine.

**The problem:** Hosted coding agents send prompts and code off the machine, and the developer wants a local model with low-latency generation instead.

**The outcome:** A running local OpenAI-compatible server at http://127.0.0.1:8000 that a coding agent client uses as its model backend.

### Steps

1. Install the precompiled package with `brew install lithos-ai/tap/lithos-metal`.
2. Start the server with `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4` and wait for compilation and warmup to finish.
3. Install your client separately, for example Claude Code or Codex, in a second terminal.
4. Launch the client through Lithos Metal with `lithos-metal claude` or `lithos-metal codex`.
5. Send a short test prompt in the client and confirm a response comes back from the local server.

### Prompt to paste into your agent

```text
Using the lithos-metal repo at https://github.com/lithos-ai/lithos-metal, start the server with `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4` on this Mac, then run `lithos-metal <client>` where <client> is the coding agent you want to use. Report the served model name and the first reply to a one-line test prompt. Verify that http://127.0.0.1:8000 answers before reporting success.
```

### Caveat

The README says one generation runs at a time, and the first start includes model download, weight preparation, compilation, and warmup, so expect a slow first run.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/local-coding-agent-backend/

## How to call the local model from scripts over HTTP with Lithos Metal

Repo: [lithos-ai/lithos-metal](https://github.com/lithos-ai/lithos-metal) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Send chat requests to the local server from your own code, with optional streaming.

**Who it is for:** A developer building a script or small app that needs LLM text generation on their own Mac.

**The problem:** The developer wants to call a model from code without a cloud account, and needs a predictable HTTP endpoint for it.

**The outcome:** A script that gets a completion from the local model through the Chat Completions endpoint, streaming or not.

### Steps

1. Install the package with `brew install lithos-ai/tap/lithos-metal`.
2. Start the server with `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4`.
3. Send a chat request with `curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"nvidia/Qwen3.8-27B-NVFP4","messages":[{"role":"user","content":"Explain how a GPU megakernel works."}],"max_completion_tokens":256,"stream":true}'`. (inferred, check the README)
4. Point your script's OpenAI-compatible client at http://127.0.0.1:8000 and use the same model name.

### Prompt to paste into your agent

```text
Using the lithos-metal repo at https://github.com/lithos-ai/lithos-metal, with the server running via `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4`, write a short script that sends <your prompt> to http://127.0.0.1:8000/v1/chat/completions and prints the reply. Expected output is the model's text reply. Verify the script receives an HTTP 200 response before reporting success.
```

### Caveat

The README says Responses is stateless and tools run in the client, so the script must send the full conversation each time and handle tool calls itself.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/call-local-api-from-scripts/

## How to compare speculative and target-only generation with Lithos Metal

Repo: [lithos-ai/lithos-metal](https://github.com/lithos-ai/lithos-metal) | Difficulty: Advanced | Time: 1 hour 30 minutes | Basis: From the README

Run the same model with DSpark drafting on and off to see how the setting changes output speed.

**Who it is for:** A performance-minded developer who wants to check whether DSpark speculative decoding helps on their own Mac and workload.

**The problem:** The README gives projections, not end-to-end results for every machine, so the developer needs to measure the difference locally.

**The outcome:** Two server runs, one with the default DSpark draft and one with target-only generation, with timings recorded for the same prompt.

### Steps

1. Install the package with `brew install lithos-ai/tap/lithos-metal`.
2. Start the default server with `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4` and time a fixed prompt with the curl call from the README.
3. Stop the server and restart it with `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4 --no-draft`. (inferred, check the README)
4. Repeat the same timed prompt and record the tokens per second for both runs.

### Prompt to paste into your agent

```text
Using the lithos-metal repo at https://github.com/lithos-ai/lithos-metal, run the same fixed prompt against `lithos-metal serve --model nvidia/Qwen3.8-27B-NVFP4` and again with `--no-draft`. Report wall-clock time and generated token count for each run in a two-row table. Verify both runs used the same prompt and max_completion_tokens before comparing.
```

### Caveat

The README warns that its own speed figures exclude drafting and rewind and are not end-to-end comparisons, so your timings only apply to your machine, model, and prompt.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/compare-draft-and-target-only-generation/

## How to build Lithos Metal from source for development with Lithos Metal

Repo: [lithos-ai/lithos-metal](https://github.com/lithos-ai/lithos-metal) | Difficulty: Advanced | Time: 2 hours | Basis: From the README

Compile the native runtime from the repo and install an editable copy for changes.

**Who it is for:** A contributor or engineer who wants to modify the engine, add a model, or test changes to the Metal kernels.

**The problem:** The precompiled Homebrew package does not cover local changes, so the developer needs a source build with the native runtime compiled.

**The outcome:** An editable install of Lithos Metal that reports its version from the local source tree.

### Steps

1. Install Xcode Command Line Tools and Python 3.12, then run `git clone https://github.com/lithos-ai/lithos-metal.git` and `cd lithos-metal`.
2. Create and activate a virtual environment with `python3.12 -m venv .venv` and `source .venv/bin/activate`.
3. Install the editable development build with `pip install -e '.[serve,dev]'`.
4. Check the build with `lithos-metal --version`, then read CONTRIBUTING.md and docs/design/extensions.md before adding a model or operation.

### Prompt to paste into your agent

```text
Using the lithos-metal repo at https://github.com/lithos-ai/lithos-metal, clone it, create a Python 3.12 virtual environment, and run `pip install -e '.[serve,dev]'`. Expected output is a successful install and a version string from `lithos-metal --version`. Verify the command prints a version before reporting success.
```

### Caveat

The README says the Python import namespace remains monolith for compatibility, so code imports will not use the lithos_metal name.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/lithos-ai-lithos-metal/build-from-source-for-development/