DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to train a mixture-of-experts model across 8 GPUs with Halo

Use Expert Parallelism on a MoE model to spread experts over 8 GPUs with one launch command.

Who it is forA researcher or platform engineer training a MoE model such as Qwen3.5-35B-A3B who needs more throughput than a single GPU gives.

The problemMixture-of-experts models are too large or slow to train on one GPU, and writing distributed expert code by hand is error-prone.

The outcomeAn SFT run that shards experts across 8 GPUs and writes a checkpoint that can be merged and exported.

What you need first

  • Docker installed on a host with an NVIDIA GPU (Hopper H100/H200 or Blackwell B200/B300 images are published)
  • Network access to public.ecr.aws to pull the Halo image anonymously
  • A Hugging Face model or the example configs in the examples/ folder
  • Enough GPU memory for the chosen model and method (the README shows single-GPU LoRA and 8-GPU runs)

Steps

  1. Pull the Blackwell or Hopper image as described in the installation section, for example docker pull public.ecr.aws/whitecircle/halo:hopper.
  2. Preview the 8-GPU launch: halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8 --dry-run. (inferred, check the README)
  3. Run the job with expert parallelism set to 8: halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8 --expert_parallel_size=8. (inferred, check the README)
  4. If the run saves expert shards, merge them with halo run merge-ep-shards --input_dir <shard_dir> --output_dir <merged_dir>. (inferred, check the README)

Prompt for your agent

In the whitecircle/halo repo, launch the MoE SFT example with `halo launch sft <target-config> -n 8 --expert_parallel_size=8` inside the Halo image on an 8-GPU node. Expected output: a completed run with expert shards in the output directory. Verify with `--dry-run` that the EP setting appears in the printed command before the real run.

Caveat. The README says EP combines only with CP, TP, or ETP and other pairings are rejected at config time, so keep the parallelism flags to those combinations.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page