# AI use cases for MLX LM (ml-explore/mlx-lm) Repo: [ml-explore/mlx-lm](https://github.com/ml-explore/mlx-lm) | Use cases: 3 | Page: https://dedupedai.com/use-cases/ml-explore-mlx-lm/ MLX LM is a Python package for generating text and fine-tuning large language models on Apple silicon with MLX. It runs thousands of models from the Hugging Face Hub with a single command, quantizes and uploads models, supports low-rank and full fine-tuning including quantized models, distributed inference and fine-tuning with mx.distributed, and tools for long prompts such as a rotating KV cache and prompt caching. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Chat with an open LLM locally on your Mac: https://dedupedai.com/use-cases/ml-explore-mlx-lm/chat-with-a-local-llm-on-a-mac/ - Quantize a model and publish it for MLX: https://dedupedai.com/use-cases/ml-explore-mlx-lm/quantize-and-publish-a-model-for-mlx/ - Ask many questions about one long document quickly: https://dedupedai.com/use-cases/ml-explore-mlx-lm/ask-many-questions-of-a-long-document-with-prompt-caching/ ## How to chat with an open LLM locally on your Mac with MLX LM Repo: [ml-explore/mlx-lm](https://github.com/ml-explore/mlx-lm) | Difficulty: Beginner | Time: 15 minutes | Basis: From the README Install MLX LM and generate text or open a chat REPL with any MLX-compatible model from the Hub. **Who it is for:** A Mac user who wants a private local assistant without a cloud API. **The problem:** Cloud chat sends your data off the machine and costs money per token. **The outcome:** A local chat REPL running a quantized model on Apple silicon. ### Steps 1. Install with `pip install mlx-lm` or `conda install -c conda-forge mlx-lm` 2. Generate text with `mlx_lm.generate --prompt "How tall is Mt Everest?"` 3. Start a chat REPL that keeps context for its lifetime with mlx_lm.chat 4. Pick another model with --model, for example `mlx_lm.generate --model mistralai/Mistral-7B-Instruct-v0.3 --prompt "hello"` 5. See all options with `mlx_lm.generate -h` ### Prompt to paste into your agent ```text Using ml-explore/mlx-lm on my Mac, install it, run mlx_lm.chat with a model from mlx-community that fits in GB, and tell me the tokens per second it reports. ``` ### Caveat The default model is mlx-community/Llama-3.2-3B-Instruct-4bit; bigger models need more unified memory. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/ml-explore-mlx-lm/chat-with-a-local-llm-on-a-mac/ ## How to quantize a model and publish it for MLX with MLX LM Repo: [ml-explore/mlx-lm](https://github.com/ml-explore/mlx-lm) | Difficulty: Intermediate | Time: 30 minutes | Basis: From the README Convert a Hugging Face model to a 4-bit MLX model and optionally upload it to the Hub. **Who it is for:** A developer who wants a smaller, faster version of a model that has no MLX build yet. **The problem:** Full precision models are too large to run comfortably on most Macs. **The outcome:** A quantized MLX model saved locally in mlx_model, and optionally uploaded to your Hub repo. ### Steps 1. Install with `pip install mlx-lm` 2. Quantize from the command line with `mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q` 3. Add --upload-repo with your target repo name to publish the result to Hugging Face 4. Or do it in Python with convert(repo, quantize=True, upload_repo=upload_repo) 5. Run the converted model with mlx_lm.generate --model pointing at it ### Prompt to paste into your agent ```text With ml-explore/mlx-lm, quantize to 4-bit using mlx_lm.convert -q, test it with mlx_lm.generate, and upload it to with --upload-repo. ``` ### Caveat This card is written from the ml-explore/mlx-lm README, not from a test run by Deduped. Before you start, make sure you have a Mac with Apple silicon. Check the repo for newer instructions before you rely on these steps. Risk, credentials: Uploading uses your Hugging Face credentials and publishes the model; check the original model's license allows redistribution. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/ml-explore-mlx-lm/quantize-and-publish-a-model-for-mlx/ ## How to ask many questions about one long document quickly with MLX LM Repo: [ml-explore/mlx-lm](https://github.com/ml-explore/mlx-lm) | Difficulty: Intermediate | Time: 20 minutes | Basis: From the README Cache a long prompt once with mlx_lm.cache_prompt and reuse it for each new question. **Who it is for:** Someone querying the same long report or codebase dump many times on a Mac. **The problem:** Re-reading a long context for every question is slow and memory hungry. **The outcome:** Fast follow-up answers that reuse a saved prompt cache file. ### Steps 1. Install with `pip install mlx-lm` 2. Cache the document: pipe prompt.txt into mlx_lm.cache_prompt with --model, --prompt - and --prompt-cache-file mistral_prompt.safetensors, as in the README 3. Ask a question with mlx_lm.generate using --prompt-cache-file mistral_prompt.safetensors and your prompt; the model is read from the cache 4. Lower peak memory while reading long prompts with --prefill-step-size 5. Use the rotating KV cache with --max-kv-size for very long generations ### Prompt to paste into your agent ```text Using ml-explore/mlx-lm, cache with mlx_lm.cache_prompt into a safetensors file, then answer these questions one by one with mlx_lm.generate --prompt-cache-file: . ``` ### Caveat Smaller --max-kv-size values save RAM but lower quality, as the README warns. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/ml-explore-mlx-lm/ask-many-questions-of-a-long-document-with-prompt-caching/