platform. WebGPU is needed, and low end phones or laptops may be too slow or run out of memory for 8B models.
Who it is forA web developer adding an assistant to a site or internal tool.
The problemHosted LLM APIs cost money per token and send user text to a server.
The outcomeA working chat reply generated locally in the browser.
What you need first
- A browser with WebGPU support on a device with a capable GPU
- Node with npm, yarn or pnpm: `npm install @mlc-ai/web-llm`, or import from the jsdelivr CDN
- Bandwidth and storage for the first model download, which the README warns can take a significant amount of time
Steps
- Install with
npm install @mlc-ai/web-llm - Import CreateMLCEngine and create an engine for Llama-3.1-8B-Instruct-q4f32_1-MLC with an initProgressCallback to show loading progress
- Call engine.chat.completions.create with your messages
- Read reply.choices[0].message and reply.usage
Prompt for your agent
Using mlc-ai/web-llm, build a minimal page that loads Llama-3.1-8B-Instruct-q4f32_1-MLC with a progress bar and answers questions with engine.chat.completions.create.
Caveat. The README says the model parameter in chat.completions.create is ignored; choose the model in CreateMLCEngine or engine.reload.
Use this in your AI agent
Ask your agent
Using the Deduped MCP, get the setup steps for mlc-ai/web-llm and "Add a private chatbot that runs in the browser"
Add Deduped to Claude Code once
claude mcp add --transport http deduped https://dedupedai.com/mcp
Every Monday: the 5 hottest repos and exactly how to use them. Free, one email a week, unsubscribe anytime.
Get them freeShare: X LinkedIn Hacker News Email Copy link
Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer