DEDUPEDWho it is forA researcher or engineer who is considering a Mixture of Experts design and needs reference code from models that the README lists as using it.
The problemMixture of Experts layers are hard to understand from papers alone, and the routing details differ between model families.
The outcomeA written note on how expert layers are structured in the two listed models, with the code locations for each part.
What you need first
- Python, as listed for the repo
- PyTorch, as stated in the repo description
- Basic familiarity with transformer attention and normalization layers
Steps
- Find the rows marked with a check in the Mixture of Experts column, including DeepSeek v3 and Llama 4 Maverick.
- Open the deepseekv3 and llama4_maverick folders linked from the table.
- Locate the expert layer and the code that picks which experts run for each token.
- Write down the number of experts and the routing logic as each folder implements them.
- Compare the two notes and list where the designs match and where they differ.
Prompt for your agent
In the repo YeonwooSung/llm-architecture-from-scratch, open the deepseekv3 and llama4_maverick folders and describe how each one implements its Mixture of Experts layer. Output a comparison table of expert structure and routing. Verify each claim against the code in the folder before reporting.
Caveat. The README marks only some models with Mixture of Experts, and the in-progress models listed at the bottom are not yet in the table.
Get 5 new use cases every morning. Free, one short email, unsubscribe anytime.
Get them freeShare: X LinkedIn Hacker News Email Copy link
Disclaimer
Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page