DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to serve Spark-X2.6-4B with vLLM for an OpenAI-style API with Spark X2 6

Deploy the model with the official vLLM Docker image and prefix caching turned on.

Who it is forAn ML platform engineer who already runs vLLM and wants to add Spark-X2.6-4B as another served model.

The problemTeams that standardize on vLLM need a documented way to load a new model into the same serving stack.

The outcomeA vLLM container serving the model on port 30000 with the served name spark25, ready for chat requests.

What you need first

  • A machine with an NVIDIA GPU or Ascend NPU for the server use cases, or enough local memory for the Transformers example
  • Docker installed for the SGLang and vLLM use cases
  • Python with pip for the Transformers use case, using transformers==4.57.1
  • Local copy of the model checkpoint or network access to Hugging Face, ModelScope, or Ollama to download it

Steps

  1. Run export MODEL_PATH=/absolute/path/to/Spark-X2.6-4B to point at your local checkpoint.
  2. Run docker run --rm --gpus all --ipc=host -p 30000:30000 -v "$MODEL_PATH:/models/Spark-X2.6-4B:ro" vllm/vllm-openai:latest --model /models/Spark-X2.6-4B --port 30000 --trust-remote-code --served-model-name spark25 --tensor-parallel-size 1 --gpu-memory-utilization 0.7 --enable-prefix-caching --chat-template /models/Spark-X2.6-4B/chat_template.jinja. (inferred, check the README)
  3. Wait for the server to load, then send a chat request to port 30000 using the served model name spark25.
  4. Check the reply text in the JSON response before connecting the endpoint to other tools.

Prompt for your agent

Using the Spark-X2.6 README vLLM NVIDIA deployment block, start vllm/vllm-openai:latest with the model at <MODEL_PATH> mounted and served as spark25 on port 30000. Send <your prompt> to the chat completions endpoint and return the reply. Verify that the response contains a non-empty message content field.

Caveat. The vLLM example serves the model as spark25 while the SGLang example uses spark2.6, so use the name your own server reports in requests.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page