DEDUPED Hot AI repos. Real use cases. What people build. Get the Monday 5

BEST AI REPOS BY TASK

Best open-source AI repos for running LLMs locally (2026)

8 repos compared · updated 2026-10-27

What is the best open-source repo for running LLMs locally?

The best open-source AI repos for running LLMs locally right now are llama.cpp, Open WebUI and vLLM. Start with llama.cpp for one machine and quantized GGUF models, add Open WebUI when you want a ChatGPT-style chat in the browser, and pick vLLM when you need to serve a model to many users through an OpenAI-compatible API.

Comparison

RepoWhat it doesStarsStart here
llama.cpp
ggml-org/llama.cpp · C++		llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models locally or in the cloud, with quantization, many GPU backends, a CLI and an OpenAI-compatible server.132,010Run an open model locally in one command Beginner, 10 minutes
Open WebUI
open-webui/open-webui · Python		Open WebUI is a self-hosted AI platform with a chat interface for Ollama and OpenAI-compatible APIs that can run entirely offline.156,670Run a private ChatGPT-style app for your local models Beginner, 10 minutes
vLLM
vllm-project/vllm · Python		vLLM is a fast, easy-to-use library for LLM inference and serving, with an OpenAI-compatible API server.95,106Serve an open LLM behind an OpenAI-compatible API Intermediate, 1 hour
LocalAI
mudler/LocalAI · Go		LocalAI is an open-source AI engine that runs LLMs, vision, voice, image and video models on any hardware, with no GPU required.49,983Self-host an OpenAI-compatible API on your own server Beginner, 30 minutes
Unsloth
unslothai/unsloth · Python		Unsloth is a desktop app and toolkit to run and train LLMs and diffusion models locally, supporting GGUF and MLX models, with installers for Windows, macOS, Linux and Docker.n/aRun and fine-tune open models from a desktop app Intermediate, 1 hour
AirLLM
lyogavin/airllm · Jupyter Notebook		AirLLM cuts inference memory so very large language models run on a single small GPU without quantization, distillation or pruning, by streaming model layers instead of loading the whole model.40,782Run a 70B-class model on a single 4GB GPU Intermediate, 1 hour
Strata
Niko1221/Strata · C++		Strata runs large open models such as Qwen3.8-Flash-Next, Qwen3.6-35B-A3B and DeepSeekV4.1-Flash on your own computer, as a desktop app or a terminal app, with no account.33,669Run a frontier-size open model locally from the terminal Beginner, 1 hour
llmfit
AlexsJones/llmfit · Rust		llmfit inspects your CPU, RAM, GPU and VRAM and recommends which open-source LLMs and quantizations your hardware can comfortably run, with a TUI, JSON output and real benchmark runs.n/aFind which open models your machine can run Beginner, 5 minutes

When to pick which

llama.cpp ggml-org/llama.cpp

llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models locally or in the cloud, with quantization, many GPU backends, a CLI and an OpenAI-compatible server.

Best for: Developers who want to run open models locally on laptops, workstations or servers with minimal setup.

Needs: macOS, Linux or Windows; Enough RAM or VRAM for the chosen GGUF model

Start here: Run an open model locally in one command Beginner, 10 minutes

  1. Install: curl -LsSf https://llama.app/install.sh | sh (Windows PowerShell: irm https://llama.app/install.ps1 | iex)
  2. Run a small model first: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
  3. Pick a quant with a suffix, for example ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M; without one it uses Q4_K_M or the first file in the repo
  4. Set a context size with -c (0 means use the model's own)

All 3 use casesGitHub

Open WebUI open-webui/open-webui

Open WebUI is a self-hosted AI platform with a chat interface for Ollama and OpenAI-compatible APIs that can run entirely offline. It adds local RAG over your documents with hybrid search, web search from many providers, custom models and agents with tools and knowledge, voice and video calls, role-based access control for teams, and SQLite or PostgreSQL storage. It installs with pip, Docker or Kubernetes.

Best for: Individuals and teams who want a ChatGPT-style interface for local or cloud models on their own server.

Needs: Docker, or Python with pip; Ollama for local models, or an OpenAI-compatible API key; A GPU is optional (the :cuda image uses it)

Start here: Run a private ChatGPT-style app for your local models Beginner, 10 minutes

  1. With Docker: docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
  2. Or with Python: pip install open-webui, then open-webui serve
  3. Open http://localhost:3000 and create the admin account
  4. Pick an Ollama model, or add an OpenAI-compatible API in settings

All 3 use casesGitHub

vLLM vllm-project/vllm

vLLM is a fast, easy-to-use library for LLM inference and serving, with an OpenAI-compatible API server.

Best for: Engineers and ML teams who need to run open-weight language models on their own GPUs or CPUs at high throughput.

Needs: Python with uv or pip installed; A supported NVIDIA, AMD or Intel GPU, or an x86, ARM or PowerPC CPU; A Hugging Face model you are allowed to download and use

Start here: Serve an open LLM behind an OpenAI-compatible API Intermediate, 1 hour

  1. Install vLLM with uv pip install vllm.
  2. Run vllm download-kernels to fetch precompiled kernels for faster startup.
  3. Start the server with vllm serve <model-id>; the README mentions an OpenAI-compatible API server but does not show this exact command, so check the Quickstart.
  4. Send one chat request to the server from your app using an OpenAI-style client.

All 4 use casesGitHub

LocalAI mudler/LocalAI

LocalAI is an open-source AI engine that runs LLMs, vision, voice, image and video models on any hardware, with no GPU required. Each backend (llama.cpp, vLLM, whisper.cpp, stable-diffusion, MLX and more) is pulled only when a model needs it, and every backend sits behind drop-in OpenAI, Anthropic and ElevenLabs compatible APIs, with multi-user auth, quotas and built-in agents with tools, RAG and MCP.

Best for: Developers and teams who want a private, self-hosted replacement for cloud AI APIs that existing OpenAI or Anthropic clients can talk to.

Needs: Docker or Podman, or the macOS app (not signed by Apple); Optional GPU: NVIDIA, AMD, Intel, Apple Silicon or Vulkan; CPU-only works; Disk space for the models you pull

Start here: Self-host an OpenAI-compatible API on your own server Beginner, 30 minutes, works with claude-code, codex, cursor

  1. Start the CPU image: docker run -ti --name local-ai -p 8080:8080 localai/localai:latest
  2. Or with an NVIDIA GPU: docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12
  3. Load a model, for example local-ai run llama-3.2-1b-instruct:q4_k_m
  4. Point your app's OpenAI base URL at http://localhost:8080

All 3 use casesGitHub

Unsloth unslothai/unsloth

Unsloth is a desktop app and toolkit to run and train LLMs and diffusion models locally, supporting GGUF and MLX models, with installers for Windows, macOS, Linux and Docker.

Best for: Developers and ML hobbyists who want to run or fine-tune open models on their own hardware.

Needs: Windows, macOS or Linux, or Docker; A GPU for training

Start here: Run and fine-tune open models from a desktop app Intermediate, 1 hour

  1. Install: brew install --cask unsloth, or winget install --id UnslothAI.Unsloth -e on Windows, or curl -fsSL https://unsloth.ai/install.sh | sh
  2. Open Unsloth and run a model to check your hardware works
  3. Build a dataset from PDFs, CSVs or DOCX files with Data Recipes
  4. Start a LoRA or QLoRA fine-tune on a small model first

All 3 use casesGitHub

AirLLM lyogavin/airllm

AirLLM cuts inference memory so very large language models run on a single small GPU without quantization, distillation or pruning, by streaming model layers instead of loading the whole model. The README reports 70B models on a 4GB card, DeepSeek-V3 (671B) on about 12GB and Kimi K3 on under 4GB, all through one AutoModel API, plus optional 4-bit or 8-bit compression for speed and new training support for huge models on small VRAM.

Best for: Developers and researchers with a modest GPU who want to experiment with very large open models without renting big cloud machines.

Needs: Python and pip; A CUDA GPU (a few GB of VRAM) or a Mac with mlx and torch for Llama-style models; Lots of disk for split layer shards; some models need specific transformers versions; Optional: bitsandbytes for compression

Start here: Run a 70B-class model on a single 4GB GPU Intermediate, 1 hour, works with claude-code, codex, cursor

  1. Run pip install airllm
  2. In Python, from airllm import AutoModel and load a model with AutoModel.from_pretrained("Qwen/Qwen3-32B") or another repo ID
  3. Tokenize your prompt with model.tokenizer as in the Quickstart
  4. Call model.generate with max_new_tokens and decode the output

All 3 use casesGitHub

Strata Niko1221/Strata

Strata runs large open models such as Qwen3.8-Flash-Next, Qwen3.6-35B-A3B and DeepSeekV4.1-Flash on your own computer, as a desktop app or a terminal app, with no account. It picks quantized model packs that fit your hardware, reads what does not fit in memory from your SSD, and exposes OpenAI- and Anthropic-compatible APIs so Claude Code, Codex CLI, OpenCode and other tools can use the local model.

Best for: Developers and privacy minded users with a recent NVIDIA, AMD, Intel Arc or Apple Silicon machine who want strong local models without cloud costs.

Needs: Windows, Linux or macOS on supported hardware (NVIDIA RTX 20 to 50, recent AMD Radeon, Intel Arc on Linux, Apple Silicon M1 or newer); Plenty of free disk space; DeepSeekV4.1 sets need about 245 to 280 GB; The model packs download on first run

Start here: Run a frontier-size open model locally from the terminal Beginner, 1 hour

  1. On Linux or Mac run curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
  2. On Windows run the PowerShell installer from the README, then open a new terminal
  3. Run strata launch and follow the prompts to download and start a model
  4. Check the Models section and docs to choose a pack that fits your memory

All 3 use casesGitHub

llmfit AlexsJones/llmfit

llmfit inspects your CPU, RAM, GPU and VRAM and recommends which open-source LLMs and quantizations your hardware can comfortably run, with a TUI, JSON output and real benchmark runs.

Best for: Developers deciding which local models to download for their machine.

Needs: macOS, Windows, Linux or Docker

Start here: Find which open models your machine can run Beginner, 5 minutes

  1. Install: brew install llmfit, winget install --exact --id AlexsJones.llmfit, or uv tool install -U llmfit
  2. Or try it without installing: uvx llmfit
  3. Run llmfit to open the interactive TUI; your hardware is detected and shown at the top
  4. Press / to filter models by name, family or quantization

All 3 use casesGitHub

Use cases for running LLMs locally

Questions

What is the best open-source repo for running LLMs locally?

The best open-source AI repos for running LLMs locally right now are llama.cpp, Open WebUI and vLLM. Start with llama.cpp for one machine and quantized GGUF models, add Open WebUI when you want a ChatGPT-style chat in the browser, and pick vLLM when you need to serve a model to many users through an OpenAI-compatible API.

Which repo for running LLMs locally is easiest to start with?

llama.cpp: its starter use case "Run an open model locally in one command" is rated Beginner and takes about 10 minutes.

How do I get started with llama.cpp?

Install: curl -LsSf https://llama.app/install.sh | sh (Windows PowerShell: irm https://llama.app/install.ps1 | iex); Run a small model first: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF; Pick a quant with a suffix, for example ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M; without one it uses Q4_K_M or the first file in the repo. Full steps and a copy-paste agent prompt: https://dedupedai.com/use-cases/ggml-org-llama-cpp/chat-with-local-model-from-hugging-face/

Are these running LLMs locally repos free?

They are open-source projects on GitHub (check each license; some, like fair-code projects, limit commercial use). Some also sell a hosted version or need a paid model API key; the requirements for each repo are listed on this page.

How is this list of running LLMs locally repos chosen?

Deduped tracks the hottest AI repos on GitHub and writes step-by-step use cases from each README. For this page we picked the repos that do this job, ordered by fit and then GitHub stars. Repos we only inferred from a short README, or that we do not index, are left out. The page is rebuilt with the site; stars are as of the last build in 2026.

More answers

For AI agents: Markdown version · JSON · MCP server · llms.txt

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer