# AI use cases for LLM Architecture From Scratch (YeonwooSung/llm-architecture-from-scratch)

Repo: [YeonwooSung/llm-architecture-from-scratch](https://github.com/YeonwooSung/llm-architecture-from-scratch) | Use cases: 3 | Page: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/

A learning repo of PyTorch implementations of modern LLM architectures, with a table comparing each model's normalization, positional encoding, attention type, and use of Mixture of Experts.

This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL.

- Compare how current LLM architectures are built: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/compare-llm-architecture-designs/
- Trace attention variants from code to table rows: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/study-attention-variants-in-code/
- Study Mixture of Experts layers in two models: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/study-mixture-of-experts-routing/

## How to compare how current LLM architectures are built with LLM Architecture From Scratch

Repo: [YeonwooSung/llm-architecture-from-scratch](https://github.com/YeonwooSung/llm-architecture-from-scratch) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Use the Implemented table to contrast normalization, positional encoding, and attention across models.

**Who it is for:** An ML engineer or student who has read about transformers and wants to see concrete design differences between recent open models before choosing or explaining one.

**The problem:** Architecture descriptions in papers and blog posts are scattered, and it is hard to line up choices like RMSNorm versus LayerNorm or RoPE versus learned absolute positions.

**The outcome:** A short side-by-side comparison of at least two models, with each design choice tied to a folder of code in the repo.

### Steps

1. Open https://github.com/YeonwooSung/llm-architecture-from-scratch and find the Implemented table.
2. Note the Normalization, Positional Encoding, Attention, and Mixture of Experts columns for each row.
3. Pick two rows with different attention types, for example Llama3-8B and DeepSeek v3.
4. Open the linked folders for both models and read the model code side by side.
5. Write a one-page comparison that cites the table cell and the code location for each design choice.

### Prompt to paste into your agent

```text
Using the repo YeonwooSung/llm-architecture-from-scratch, compare the rows for <model A> and <model B> in the Implemented table. Output a table of normalization, positional encoding, attention type, and Mixture of Experts use. Verify each cell against the README row before reporting.
```

### Caveat

The README is a table of design choices and does not describe training results, so the comparison covers architecture only.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/compare-llm-architecture-designs/

## How to trace attention variants from code to table rows with LLM Architecture From Scratch

Repo: [YeonwooSung/llm-architecture-from-scratch](https://github.com/YeonwooSung/llm-architecture-from-scratch) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README

Map the attention types in the table, such as GQA, sliding window, and MLA, to their implementations.

**Who it is for:** A developer preparing a talk or internal review on efficient attention who needs working code examples for each variant.

**The problem:** Terms like Grouped-Query Attention, sliding window, and Multi-head Latent Attention are easy to mix up without seeing how each one is written.

**The outcome:** A map from each attention label in the table to the folder that implements it, with notes on the main code differences.

### Steps

1. Open the Attention column of the Implemented table in the repo README.
2. Group the rows by attention label: Multi-Head Self-Attention, Grouped-Query Attention, Sliding window + GQA, Multi-head Latent Attention, mLSTM, and GQA.
3. Open one folder per label, starting with the gpt2-xl and llama3-8b folders.
4. Compare how query, key, and value heads are shaped in each folder.
5. Record the differences in a short table for the talk or review.

### Prompt to paste into your agent

```text
In the repo YeonwooSung/llm-architecture-from-scratch, find the folder for <attention type> from the Attention column and summarize how its heads are built. Output a bullet list of code-level differences from Grouped-Query Attention. Verify the folder name matches the README link before reporting.
```

### Caveat

The repo does not document setup or run commands, so check each folder for its own instructions before running any code.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/study-attention-variants-in-code/

## How to study Mixture of Experts layers in two models with LLM Architecture From Scratch

Repo: [YeonwooSung/llm-architecture-from-scratch](https://github.com/YeonwooSung/llm-architecture-from-scratch) | Difficulty: Advanced | Time: 1 hour 30 minutes | Basis: From the README

Read the MoE implementations in the DeepSeek v3 and Llama 4 Maverick folders to see how experts are used.

**Who it is for:** A researcher or engineer who is considering a Mixture of Experts design and needs reference code from models that the README lists as using it.

**The problem:** Mixture of Experts layers are hard to understand from papers alone, and the routing details differ between model families.

**The outcome:** A written note on how expert layers are structured in the two listed models, with the code locations for each part.

### Steps

1. Find the rows marked with a check in the Mixture of Experts column, including DeepSeek v3 and Llama 4 Maverick.
2. Open the deepseekv3 and llama4_maverick folders linked from the table.
3. Locate the expert layer and the code that picks which experts run for each token.
4. Write down the number of experts and the routing logic as each folder implements them.
5. Compare the two notes and list where the designs match and where they differ.

### Prompt to paste into your agent

```text
In the repo YeonwooSung/llm-architecture-from-scratch, open the deepseekv3 and llama4_maverick folders and describe how each one implements its Mixture of Experts layer. Output a comparison table of expert structure and routing. Verify each claim against the code in the folder before reporting.
```

### Caveat

The README marks only some models with Mixture of Experts, and the in-progress models listed at the bottom are not yet in the table.

Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/

URL: https://dedupedai.com/use-cases/yeonwoosung-llm-architecture-from-scratch/study-mixture-of-experts-routing/