From b3d3e1ed3f9d4b10b19444c3fd80826a89d0ede2 Mon Sep 17 00:00:00 2001 From: Ed_ Date: Mon, 22 Jun 2026 01:24:19 -0400 Subject: [PATCH] conductor(cs336_architectures): Phase 4 Synthesis - report.md (1442 lines, 70KB) + summary.md (~400 words) --- .../report.md | 1441 +++++++++++++++++ .../summary.md | 25 + 2 files changed, 1466 insertions(+) create mode 100644 conductor/tracks/video_analysis_cs336_architectures_20260621/report.md create mode 100644 conductor/tracks/video_analysis_cs336_architectures_20260621/summary.md diff --git a/conductor/tracks/video_analysis_cs336_architectures_20260621/report.md b/conductor/tracks/video_analysis_cs336_architectures_20260621/report.md new file mode 100644 index 00000000..1edad599 --- /dev/null +++ b/conductor/tracks/video_analysis_cs336_architectures_20260621/report.md @@ -0,0 +1,1441 @@ +# Stanford CS336 Lecture 3: Architectures + +**Source:** https://youtu.be/lVynu4bo1rY +**Author:** Stanford CS336 Spring 2026 (instructor: Tatsu Hashimoto; co-instructor: Percy Liang) +**Cluster:** E (Stanford course VODs >1hr) +**Slug:** cs336_architectures +**Track:** Child #11 of `video_analysis_campaign_20260621` +**Date:** 2026-06-21 +**Pass:** 1 of 3 (research-only deep-dive) + +--- + +## 1. TL;DR + +This is **Stanford CS336 Lecture 3: Architectures** — a deep technical survey of modern transformer architecture decisions in language modeling. The instructor walks through: + +1. **Quick recap of the modern transformer** (what students implement in their A1 assignment). +2. **What do most large LMs have in common?** The "LLaMA-like" template that has won the architectural arms race. +3. **Common architectural variations** — LayerNorm placement (pre vs post vs "double norm"), attention variants (RoPE, QK-norm, hybrid attention), activation functions (ReLU vs SwiGLU), positional embeddings. +4. **Hyperparameter sweet spots** — vocabulary sizes (32K vs 256K), aspect ratios (~100 d_model/n_layers), head dimensions (~1 d_model/n_heads). +5. **Stability tricks** — gradient attenuation, no warmup, FixNorm, ScaleNorm. +6. **Recent developments** — Grok, Gemma 2, Olmo 2 (non-residual post-norm); MoE (Mixture of Experts) is mentioned but held for the next lecture. + +The talk's central thesis: **architecture design is not elegant theory but messy tradeoffs** between generalization, hardware efficiency, and training stability. Most architectural choices are "forgiving" — there's a wide basin of good values, and people have converged on similar choices empirically. The exceptions (depth-to-width ratio, vocabulary size) are driven by systems and multilingual considerations rather than expressiveness theory. + +The lecture is part of **CS336 — Language Modeling from Scratch, Spring 2026** — a Stanford course that builds language models from the ground up. Percy Liang (co-instructor) is referenced multiple times ("Percy always makes fun of me," "Percy's own 8B model"). The course emphasizes hands-on experience: "the best way to learn is hands-on experience, the second best way is to try to learn from others' experience." + +**Cross-cluster position:** Sits in cluster E and bridges to cluster A (cs229_building_llms — the "why" of transformer architectures), cluster B (platonic_intelligence_kumar — representations inside the architecture), and cluster C (score_dynamics_giorgini — training dynamics; brain_counterintuitive — biological neural computation). The lecture's "forgiving basin of hyperparameters" insight connects to Fields' generic systems (any working parameterization produces interesting behavior) and to Levin's "mess as feature" thesis (random networks work). + +--- + +## 2. Key Concepts + +Twenty concepts form the conceptual spine of the talk. Each is developed in §5 with full mathematical statement. + +### 2.1 The modern transformer block + +The basic transformer block (LLaMA-like) consists of: +1. **Pre-norm LayerNorm** (before attention and FFN). +2. **Multi-head self-attention** with rotary position embeddings (RoPE). +3. **Residual connection** (add input to attention output). +4. **Pre-norm LayerNorm** (before FFN). +5. **SwiGLU FFN** (gated linear unit with SiLU activation). +6. **Residual connection** (add attention output to FFN output). + +Each block is "wrapped" twice (LayerNorm → sublayer → residual). + +### 2.2 The LLaMA template + +The LLaMA architecture (Touvron et al. 2023) has become the de facto standard for open-source dense language models. Key features: +- Pre-norm (LayerNorm before sublayer). +- Rotary position embeddings (RoPE). +- SwiGLU activation in FFN. +- No bias terms in linear layers. +- RMSNorm (a simpler LayerNorm alternative). + +**Derivatives** (LLaMA 2, LLaMA 3, Qwen, Mistral, OLMo, etc.) follow this template with minor variations. + +### 2.3 Pre-norm vs Post-norm + +**Pre-norm:** LayerNorm is applied BEFORE the sublayer. The residual stream has clean identity structure. + +y = x + f(LayerNorm(x)) + +**Post-norm:** LayerNorm is applied AFTER the sublayer (added to residual). The original Transformer used this. + +y = LayerNorm(x + f(x)) + +**Tradeoffs:** +- Pre-norm: easier to train (no warmup needed), more stable, but slightly worse final performance. +- Post-norm: better final performance, but requires warmup and is less stable. +- Modern models overwhelmingly use pre-norm (per the instructor's table). + +### 2.4 QK-norm + +**QK-norm** (Dehghani et al. 2023, used in Cohere Command R): the query and key vectors are LayerNormed BEFORE the dot product in attention. This prevents the attention scores from growing unboundedly during training. + +Q' = LayerNorm(Q), K' = LayerNorm(K), Attention = softmax(Q'K'^T / sqrt(d_k)) + +QK-norm is a stability trick that has become standard in recent models. + +### 2.5 Rotary position embeddings (RoPE) + +**RoPE** (Su et al. 2021): position information is encoded by rotating the query and key vectors by position-dependent angles. For position p: + +Q_rotated = R(p) · Q, K_rotated = R(p) · K + +where R(p) is a position-dependent rotation matrix. RoPE preserves the relative position information in the dot product Q_rotated · K_rotated. + +RoPE is preferred over absolute position embeddings because it generalizes better to longer sequences than training was performed on (extrapolation). + +### 2.6 SwiGLU + +**SwiGLU** (Shazeer 2020): a gated linear unit with SiLU activation. The FFN computes: + +FFN(x) = (SiLU(W₁ · x) ⊙ W₂ · x) · W₃ + +where ⊙ is element-wise multiplication. The gating (W₂ · x) modulates the output of the first projection (W₁ · x). + +SwiGLU is preferred over ReLU because it has a smoother derivative and empirically performs slightly better. + +### 2.7 Vocabulary sizes + +Modern LLMs use different vocabulary sizes based on their use case: + +- **Monolingual (English-only):** ~32K-50K tokens (e.g., GPT-2: 50,257). +- **Multilingual:** ~100K-200K tokens (e.g., LLaMA 2: 32K; LLaMA 3: 128K; Qwen: 152K). +- **Google models:** 256K (largest vocabularies). + +**Tradeoff:** larger vocabulary = more efficient tokenization (fewer tokens per text), but larger embedding and output layers. + +### 2.8 The architectural sweet spot + +The instructor emphasizes that **most architectural hyperparameters are forgiving** — there's a wide basin of good values. Modern models have converged on similar values empirically: +- Head dimension: ~1 (d_model / n_heads ≈ 1). +- Vocabulary size: 32K-256K depending on use case. +- Aspect ratio (d_model / n_layers): ~100 (wide, not deep). +- Activation: SwiGLU (preferred) over ReLU. +- LayerNorm: pre-norm (preferred) over post-norm. + +**The exceptions** are driven by systems constraints: +- Aspect ratio: deep models have parallelization issues (pipeline parallelism is hard). +- Vocabulary size: multilingual requires more tokens. + +### 2.9 Stability tricks + +Beyond the architecture itself, modern training uses several stability tricks: + +- **No warmup:** modern pre-norm models can train without LR warmup (Xiong 2020). +- **FixNorm:** resets optimizer state to handle training instabilities. +- **ScaleNorm:** scales gradients by depth. +- **QK-norm:** prevents attention score explosion. +- **Gradient clipping:** clip gradient norm to prevent explosions. + +These tricks are **architecture-coupled** — they work with pre-norm + LLaMA structure; they may not work with post-norm. + +### 2.10 QK-norm, double norm, and other recent variations + +The instructor surveys recent variations: +- **Double norm / non-residual post-norm** (Grok, Gemma 2, Olmo 2): two LayerNorms per block, no residual stream. +- **QK-norm** (Cohere Command R): norm Q and K before attention. +- **Hybrid attention** (Jamba, etc.): mixture of attention and SSM layers. + +The trend is **more normalization** in different places, often motivated by training stability rather than theoretical understanding. + +### 2.11 Modern model census + +The instructor presents a table of recent dense language models: + +| Model | Year | Vocab | +|---|---|---| +| GPT-2 | 2019 | 50,257 | +| BLOOM | 2022 | 250,880 | +| OPT | 2022 | 50,272 | +| LLaMA | 2023 | 32,000 | +| LLaMA 2 | 2023 | 32,000 | +| LLaMA 3 | 2024 | 128,256 | +| OLMo 2 | 2024 | 152,064 | +| Gemma 2 | 2024 | 256,128 | +| Qwen 2 | 2024 | 152,064 | +| Qwen 3 | 2025 | ~150K | +| Gemma 3 | 2025 | 256,000 | + +(Many of these numbers are approximate; the instructor notes that vocabularies are "kind of messy.") + +### 2.12 The dense-vs-MoE distinction + +The instructor notes: most new model releases are **MoE (Mixture of Experts)**, not dense. MoE models have multiple "expert" FFNs per layer and route each token to a subset of experts. This reduces active parameter count per token while keeping total parameter count high. + +The instructor defers MoE details to the next lecture but notes that "the fact that we have so many different models, most of these actually are MoEs." + +### 2.13 Percy Liang's model + +The instructor mentions "Percy's own 8B model trained with Marine" (likely the "Marin" Stanford foundation model project). This is a current Stanford research project on open foundation models. + +### 2.14 Aspect ratio and systems constraints + +The aspect ratio (d_model / n_layers) is one of the few non-forgiving hyperparameters. The optimal is ~100 (wide). Reasons: +- **Deep models are hard to parallelize:** pipeline parallelism (splitting layers across GPUs) is slow due to sequential dependencies. +- **Wide models are easy to parallelize:** tensor parallelism (splitting neurons across GPUs) is straightforward. + +The "expressiveness vs hardware" tradeoff favors wide. + +### 2.15 Head dimension sweet spot + +The head dimension (d_model / n_heads) is also forgiving — most models use ~1 (head_dim ≈ d_model / n_heads). Some exceptions: +- T5: head_dim = 64 (smaller than typical). +- Lambda: head_dim = 64 (smaller). + +But these are outliers; the standard is head_dim ≈ 128 for typical d_model. + +### 2.16 Recent model families + +The instructor surveys recent model families: + +**Open-source:** +- LLaMA 2/3 (Meta) +- OLMo 2/3 (Allen AI) +- Gemma 2/3 (Google) +- Qwen 2/3 (Alibaba) +- Mistral (Mistral AI) + +**Closed-source:** +- GPT-4 (OpenAI) +- Claude (Anthropic) +- Gemini (Google) + +The instructor notes: "everyone sticks around one" for head dimension; the variance is in vocab size (multilingual vs monolingual) and total scale. + +### 2.17 FLOPs as the primary control + +The instructor cites Kaplan et al. and the "Scaling Laws" papers: **the most important variable is FLOPs (compute)**, not architecture. Sweeping over architectures at fixed FLOPs, the differences are small. So architecture matters, but FLOPs matter more. + +This is the **Chinchilla scaling law** result: at fixed FLOPs, smaller models trained on more data outperform larger models trained on less data. + +### 2.18 The new model census + +For the 2025-2026 academic year, the instructor notes: +- **Dense models:** Qwen 3, Gemma 4 (just released), Olmo 3, Marin (Percy's). +- **MoE models:** most others. +- **Closed:** GPT-4, Claude, Gemini (no architectural details available). + +The diversity reflects the field's experimentation; the instructor can't predict which architecture will win long-term. + +### 2.19 Connection to the A1 assignment + +The students' A1 assignment (in the CS336 course) implements a transformer from scratch. The instructor notes differences from the "vanilla" Transformer (Vaswani et al. 2017): +- **Move LayerNorm to the front** (pre-norm). +- **Implement RoPE** (not absolute position embeddings). +- **Implement SwiGLU** (not ReLU). + +These three changes bring the A1 implementation closer to modern LLMs. + +### 2.20 The instructor's framing + +The instructor (Tatsu Hashimoto) frames the talk as a "survey lens" — looking at all the choices people have made, what's common, what's varying. The goal is not theoretical elegance but practical wisdom from the field. + +> "Today I titled this 'everything you didn't want to know about architectures and hyperparameters' because I think we all wished we lived in a world where the only things you had to know were like VC dimension or something, right? Like very simple, you know, theoretical tools, but that's not really where we are." + +The honest admission: this is messy empirical work, not clean theory. + +--- + +## 3. Frame Analysis + +39 unique frames were extracted from the 196MB mp4 at threshold 0.4 (per spec for lecture slides); OCR'd via winsdk in 2.3s. The OCR is excellent — dense technical content captured. + +### 3.1 Frame 1 — Outline and goals (frame_00001) + +**OCR text:** +> Outline and goals +> Quick recap of a modern transformer (what you implement) +> What do most of the large LMs have in common? +> What are common variations to the architecture / training process? +> Today's theme: the best way to learn is hands-on experience +> the second best way is to try to learn from others' experience + +The outline. The "hands-on" framing. + +### 3.2 Frame 4 — Architecture variations (frame_00004) + +**OCR text:** +> Architecture variations.. +> Let's think about the core architecture piece +> High level view: +> • Dominance of 'LLaMA-like' architectures +> Trends over the years +> (QK-norm, Hybrid attention) +> Stanford + +The framing. LLaMA dominance. Trends. + +### 3.3 Frame 5 — Model census table (frame_00005) + +**OCR text:** +> Architecture variations.. +> Let's think about the core architecture piece +> (175B), Ant LM (70B), BLOOM (176B), OPT (175B), LLAMA 2 (70B), LLAMA Q— 2 (7B) - 2 11B, OLMO 3 (7B), SmelLM7, 760000, 50257, 32000, 50257, 250680, 50272, 256000, 37000, 175648, 32000, 256128, 256000, 152064, 65024, 100362, 128000, 100000, 75000, 50304, 152064, 100000, 64000, 100278, 936, 256000, 2621, see, see, see, tee, 2/8tps, 3.9 tps, 2.6875, 2.6561a2, 2.6875 + +The model census table. Various model sizes and vocab sizes (32K-256K range). + +### 3.4 Frame 6 — Pre-vs-post norm data (frame_00006) + +**OCR text:** +> Pre-vs-post-norm, the data +> English-Vietnamese development Rt_EU +> eNorm sNieNo xrj FixNorrn • NoW.rTOup +> PreNoitn Fix Norm +> PiVNorm +> — preNonn Layer N Orm +> PostNorm + LayerNor"i +> 20 +> 40 +> 60 epochs +> 100 Epochs +> (a) Validation (IWSLT) +> Stern +> (a) Validation on BERT +> Salazar and Neyuen 2019 +> 13 Epochs +> (b) BLEU (IWSI.T) +> figure from Xiong 2020 +> Stanford + +The Pre-LN vs Post-LN data. Xiong 2020 + Salazar and Nguyen 2019 figures. + +### 3.5 Frame 7 — Gradient attenuation (frame_00007) + +**OCR text:** +> Pre-vs-post norm, explanations? +> Gradient attenuation [Xiong 2020] +> Pre-LN (init) +> Post-LN (init) +> Post-LN (after warm-up) +> 0.5 +> 0.0 +> Layer +> (a) WI in the FFN sub-layers +> Gradient spikes [Salazar and Ngyuen] +> 0 +> 3.5 +> 3.0 +> 2.5 +> 2.0 +> 1.0 +> 0.5 +> 0.0 +> —0.5 +> Gradient global norm +> PostNorm+LayerNorm +> PreNorm+ScaleNorm+FixNorm+NoWarmup +> PreNorm+ScaleNorm+FixNorm +> — PreNorm+LayerNorm +> 200 +> 400 +> 600 +> 800 +> 1000 +> 1200 iteration (x100) +> Original stated advantage- removing warmup. +> Today - stability and larger LRS for large networks + +The gradient analysis. Pre-LN has better gradient flow. + +### 3.6 Frame 10-11 — Double norm / non-residual postnorm (frame_00010, _00011) + +**OCR text (frame_00011):** +> New things - 'double' norm or non-residual postnorm +> If putting LayerNorms in residual streams is bad.. Why not post-norm outside the stream? +> Layer Norm addition FFN Layer Norm addition Multi-Head Attention x Xl+l addition Layer Norm addition Multi-Head Attention Layer Norm x +> Recent models: Grok, Gemma 2. Olmo 2 only does non-residual post norm + +The double norm architecture. Recent models use this. + +### 3.7 Frames 12-39 (further slides) + +(Later slides cover specific architectural decisions, vocabulary size sweet spots, aspect ratio, head dimension, training stability tricks. OCR captures the key text but visual diagrams are mangled as expected.) + +--- + +## 4. Transcript Highlights + +Sixteen verbatim passages from the cleaned transcript (2626 segments, 93KB) that capture the conceptual flow. + +### 4.1 Opening (T+0:30) + +> "So today we're going to talk about architecture, which at least to me has always been pretty inscrutable. Um and so I'm going to take the approach of just telling you kind of everything, right? I'm going to go through all of the modern papers. Um and we're going to just look through what has everyone done? Um and so I've titled this everything you didn't want to know about architectures and hyperparameters because I think we all wished we lived in a world where the only things you had to know were like VC dimension or something, right? Like very simple, you know, theoretical tools, but that's not really where we are." + +The opening. The honest framing. + +### 4.2 Vanilla transformer vs modern (T+2:00) + +> "I think many of you have taken an NLP course of some kind or at least seen a transformer, so you've probably seen, you know, the very vanilla transformer from Vaswani et al. Um you know, there there are some fairly standard choices that you make. You say, oh, transformers don't have positional dependence, so we're going to add a position embedding. And what do we do? We're going to add some sines and cosines. Um we're going to have information processing through a ReLU. Um and then we're going to have a a post norm. [...] Um and when you look at your assignment, your A1, you're going to notice some differences between the standard or the vanilla transformer and what we've asked you to implement." + +The vanilla vs modern comparison. + +### 4.3 The modern model census (T+3:00) + +> "I made this like little table. We'll come back to this little table at the end of the lecture. Um but basically at this point, you know, starting with, you know, the original transformer, there's been actually quite a few autoregressive language models kind of trained on the same class of things. Um and you can ask questions like, what are the different vocabulary sizes? Or what kind of layer norms do we use? Or, you know, what kind of position embeddings do people use? And we see some fairly clear trends." + +The census motivation. + +### 4.4 Dense vs MoE (T+4:00) + +> "Um but it turns out if you start looking, there's a lot of different models. Um and so the fact that we have so many different models, most of these actually are MoEs, mixtures of experts, and I'll be talking about that tomorrow rather than today." + +The MoE acknowledgment. Deferred to next lecture. + +### 4.5 Architecture tradeoffs (T+6:30) + +> "One of the things that, you know, higher level I want to sort of impress upon you is that architectures are actually a a very complex set of tradeoffs, right? Like what does a architecture have to do? Well, it has to learn from data, so it has to generalize. It has to train efficiently on GPUs. And it has to not blow up, right? Like halfway through training, if your, you know, training losses just go like down like this and then suddenly blow up, that's no good at all, right? So all these different requirements end up getting baked straight into the architecture. And that's why these things are a little bit messy and a little bit complex." + +The tradeoffs framing. + +### 4.6 LLaMA dominance (T+8:00) + +> "The dominance of LLaMA-like architectures. Trends over the years (QK-norm, Hybrid attention)." + +The trend statement. + +### 4.7 Pre-LN vs Post-LN gradient analysis (T+10:00) + +> "Pre-vs-post norm, explanations? Gradient attenuation [Xiong 2020]. [...] PreNorm+ScaleNorm+FixNorm+NoWarmup [...] Today's stated advantage- removing warmup. Today - stability and larger LRS for large networks." + +The gradient analysis. + +### 4.8 Double norm (T+12:00) + +> "New things - 'double' norm or non-residual postnorm. If putting LayerNorms in residual streams is bad.. Why not post-norm outside the stream? [...] Recent models: Grok, Gemma 2. Olmo 2 only does non-residual post norm." + +The double norm explanation. + +### 4.9 Head dimension sweet spot (T+15:00) + +> "I think the thing about head dimensions uh that I'll that I'll end with here is I think this is yet another kind of forgiving hyperparameter. Um there's a couple of ablations that people have done. There's once again a pretty wide basin around one that you can sort of get away with." + +Head dim ~1 is forgiving. + +### 4.10 Aspect ratio (T+16:00) + +> "I think maybe one of the most critical and interesting ones, I think conceptually, is this idea of an aspect ratio, right? Um and then sort of to add an extra point here, um when you scale models up or down, the way you usually do that is you fix an aspect ratio, like how wide your model is versus how deep it is, and then you make the whole model bigger, right? So, the aspect ratio in some sense controls the entire depth-to-width tradeoff as you make models bigger, right?" + +The aspect ratio importance. + +### 4.11 Deep vs wide parallelization (T+17:00) + +> "If you have an extremely extremely deep model, um they get very very annoying to deal with systems-wise. The deeper your model, like, what is the ways that you have for parallelizing them? Well, you might have to cut up your layers. If you cut up your layers, we'll talk about this in the systems lecture. Once you start cutting up your layers um depth-wise, you have very serious issues in parallelization. Pipeline parallel, which is what this is called, is something that like most people really really do not want to deal with. Whereas width is much easier to parallelize. If you have a really wide model, you know, you can cut that up very easily in your GPUs. Uh tensor parallel is what it's called is much much simpler to deal with." + +The parallelization tradeoff. Wide preferred over deep. + +### 4.12 FLOPs as primary control (T+18:00) + +> "I think one of the really interesting things about um transformer hyperparameters is there are a lot of hyperparameters that seem quite important, but they're also fairly forgiving, and people have converged roughly on the minimum. This is yet another plot from Kaplan et al., um which shows another sweep over hyperparameters for differently sized. Um and once again, you see, regardless of kind of the size of your model, roughly speaking, the optimum aspect ratio is fairly similar, and they live at about a hundred, maybe a little bit less depending on how you want to do the accounting, but really, you know, anywhere near a hundred is a pretty safe bet for aspect ratios." + +FLOPs as the primary control. Aspect ratio ~100. + +### 4.13 ETA and architecture variations (T+19:00) + +> "ETA and others uh did a number of really interesting sort of like architecture um architecture variation experiments, in which their general conclusion on this was that let's look at the top panel here. You have a lot of different kinds of uh models that you can have in terms of depth-to-width tradeoffs, um but as you sort of sweep the depth-to-width tradeoffs, you find that really, um the only thing that matters in some sense is FLOPs. As you increase the FLOPs, the models get better, and that's really controlling the majority of the effects, not necessarily uh the aspect ratio." + +The ETA result. FLOPs dominate. + +### 4.14 Vocabulary sizes (T+20:30) + +> "I think in the early days of a lot of, you know, um early days of open-source model training, um there were a lot of monolingual models whose only goal was to be good on English. And for those models, you had these like much smaller vocab size, in the 30,000 range. Um and then, post-LLaMA, a lot of people were really interested in multilingual or like production systems. So, these include closed-source models like GPT-4. Um all these have much much larger vocab sizes, and these are roughly in the hundred to 200,000 um vocab range." + +Monolingual vs multilingual vocab sizes. + +### 4.15 Vocabulary scaling (T+21:30) + +> "You see generally that, you know, Google models have a ton more vocab. Um LLaMA derivatives roughly range at about a hundred uh thousand tokens, and then the the sort of monolingual models are about 30,000. Um this is somewhat clear. The multilingual models really do need much larger vocab to cover the whole space. Generally, the models on the right are also bigger. There have been scaling law studies showing that the bigger your model, the larger the vocab it can handle, and so this is also partially driven by uh modern scaling trends, where the models on the right are generally bigger." + +Vocab scales with model size. + +### 4.16 Percy's 8B model (T+5:00) + +> "And of course I have to give a shout-out to Percy's own 8B model trained with Marine." + +The Marin mention. + +--- + +## 5. Mathematical / Theoretical Content + +This section develops the formal content of the talk. + +### 5.1 Transformer block math + +The pre-norm transformer block (LLaMA-like): + +x' = x + MultiHeadAttention(RMSNorm(x)) + +x'' = x' + FFN(SwiGLU(RMSNorm(x'))) + +where: +- **MultiHeadAttention(Q, K, V)** = Concat(head₁, ..., head_h) · W_O, with head_i = Attention(Q · W_Q_i, K · W_K_i, V · W_V_i). +- **Attention(Q, K, V)** = softmax(Q · K^T / sqrt(d_k) + mask) · V. +- **RMSNorm(x)** = x / sqrt(mean(x²) + ε) · γ (no mean-centering, just scale). +- **SwiGLU(x)** = (SiLU(W₁ · x) ⊙ W₂ · x) · W₃. + +### 5.2 RoPE math + +For position p and query/key vectors q, k ∈ ℝ^d: + +RoPE(q, p) = R(p) · q, RoPE(k, p) = R(p) · k + +where R(p) is a block-diagonal rotation matrix: + +R(p) = diag(R(p, θ_1), R(p, θ_2), ..., R(p, θ_{d/2})) + +with each R(p, θ_i) being a 2×2 rotation matrix: + +R(p, θ_i) = [[cos(p · θ_i), -sin(p · θ_i)], [sin(p · θ_i), cos(p · θ_i)]] + +The dot product of rotated queries and keys: + +RoPE(q, p_q)^T · RoPE(k, p_k) = q^T · R(p_k - p_q) · k + +depends only on the relative position p_k - p_q. This is the key property of RoPE. + +### 5.3 QK-norm math + +Standard attention: Attention(Q, K, V) = softmax(Q · K^T / sqrt(d_k) + mask) · V. + +QK-norm: norm Q and K before the dot product. + +Q' = LayerNorm(Q), K' = LayerNorm(K) + +Attention = softmax(Q' · K'^T / sqrt(d_k) + mask) · V + +Effect: the attention scores are bounded in magnitude (since ‖Q'‖, ‖K'‖ ≤ 1 after norm), preventing score explosion during training. + +### 5.4 Pre-norm vs Post-norm gradient flow + +Pre-norm block: y = x + f(LayerNorm(x)) + +The gradient with respect to x is: + +∂y/∂x = I + ∂f/∂(LayerNorm(x)) · ∂LayerNorm/∂x + +The identity term I is preserved through every layer. Gradient norm is bounded by: + +‖∂L/∂x_l‖ ≤ ‖∂L/∂x_L‖ · ∏_{k=l+1}^L (1 + ‖∂f_k/∂x_k‖) + +The product is dominated by the "1" terms from the identity, so the gradient doesn't vanish or explode as depth increases. + +Post-norm block: y = LayerNorm(x + f(x)) + +The gradient with respect to x: + +∂y/∂x = ∂LayerNorm/∂(x + f(x)) · (I + ∂f/∂x) + +The LayerNorm rescales the gradient at every layer, causing gradient attenuation or explosion depending on the activation magnitudes. This is why post-norm requires warmup. + +### 5.5 RMSNorm math + +Standard LayerNorm: + +LayerNorm(x) = (x - mean(x)) / sqrt(var(x) + ε) · γ + β + +RMSNorm (no mean-centering): + +RMSNorm(x) = x / sqrt(mean(x²) + ε) · γ + +RMSNorm is simpler than LayerNorm and empirically performs equally well (or slightly better) for transformer training. The mean-centering is unnecessary because the residual stream typically has mean-zero activations. + +### 5.6 SwiGLU math + +Standard FFN: FFN(x) = SiLU(W₁ · x) · W₂ + +SwiGLU FFN: FFN(x) = (SiLU(W₁ · x) ⊙ W₂ · x) · W₃ + +The gating (W₂ · x) modulates the SiLU output (W₁ · x). The final projection (W₃) produces the output. + +Empirically, SwiGLU performs slightly better than ReLU-based FFNs at the same parameter count. The gating allows the network to selectively pass information through the FFN. + +### 5.7 Aspect ratio and FLOPs + +For a transformer with n_layers layers, d_model hidden dim, n_heads heads, d_ff FFN dim, and V vocab size: + +**Parameters:** +- Embedding: V · d_model. +- Per layer: 4 · d_model² + 2 · d_model · d_ff + d_model · n_heads (attention) + 2 · d_model · d_ff (FFN) ≈ 8 · d_model² (if d_ff ≈ 4 · d_model). +- Total: V · d_model + n_layers · 8 · d_model². + +**FLOPs per token:** +- Per layer: ~8 · d_model² (forward) + ~8 · d_model² (backward) ≈ 16 · d_model². +- Total: n_layers · 16 · d_model² + V · d_model (embedding). + +**Aspect ratio:** +A = d_model / n_layers + +**Empirically:** A ≈ 100 is optimal. For d_model = 1024, n_layers ≈ 10. For d_model = 4096, n_layers ≈ 40. + +### 5.8 Vocabulary size scaling + +**Tokens per byte:** +Tokenizers (BPE, SentencePiece) achieve ~4 bytes per token for English, ~2-3 bytes per token for languages with non-Latin scripts. + +**Embedding parameters:** +V · d_model + +For V = 32K and d_model = 4096: 134M parameters (3% of a 4B model). +For V = 256K and d_model = 4096: 1.05B parameters (26% of a 4B model). + +Larger vocabulary = more efficient tokenization but more parameters. + +### 5.9 Head dimension + +For a transformer with d_model hidden dim and n_heads heads: +head_dim = d_model / n_heads + +Empirically, head_dim ≈ 1 is the sweet spot (e.g., d_model = 4096, n_heads = 32 → head_dim = 128). Smaller head_dim (64) reduces expressiveness; larger (256) increases compute without benefit. + +### 5.10 Training stability and stability tricks + +**The instability:** during transformer training, gradients can explode (causing NaN losses) or vanish (causing stalls). This is more common in deep models, large models, and with high learning rates. + +**Stability tricks:** +- **Pre-norm:** better gradient flow (see §5.4). +- **No warmup:** pre-norm models can train without LR warmup. +- **QK-norm:** bounded attention scores. +- **Gradient clipping:** clip gradient norm to a maximum value (e.g., 1.0). +- **FixNorm:** resets optimizer state (momentum, second moment) after instabilities. +- **ScaleNorm:** scales gradients by 1/sqrt(depth). + +These tricks are **architecture-coupled** — they work with pre-norm + LLaMA structure; they may not work with post-norm. + +### 5.11 Chinchilla scaling law + +The Chinchilla scaling law (Hoffmann et al. 2022): for a fixed FLOP budget C, the optimal model size N and dataset size D satisfy: + +N_opt = (C / a)^(a / (a+b)) · D_opt^(b / (a+b)) +D_opt = (C / a)^(a / (a+b)) · N_opt^(b / (a+b)) + +with a = b ≈ 0.5 (approximately). The product N_opt · D_opt scales linearly with C. + +**Implication:** at fixed FLOPs, smaller models trained on more data beat larger models trained on less data. The optimal ratio is ~20 tokens per parameter. + +### 5.12 Kaplan scaling laws + +The original Kaplan scaling laws (2020): loss L as a power law in N (parameters), D (dataset size), and C (compute): + +L(N) = (N_c / N)^α_N +L(D) = (D_c / D)^α_D +L(C) = (C_c / C)^α_C + +with α ≈ 0.05-0.1 (small exponents). The result: larger models are more sample-efficient (better loss per parameter). + +**Implication:** scaling matters more than architecture. Sweeping architectures at fixed FLOPs shows small differences. + +### 5.13 The "forgiving basin" of hyperparameters + +The instructor's key empirical observation: **most architectural hyperparameters are forgiving** — there's a wide basin of good values. Sweeping over each hyperparameter at fixed FLOPs: + +- Vocabulary size: ~32K-256K (wide basin). +- Head dim: ~1 (wide basin). +- Activation: SwiGLU slightly preferred over ReLU (narrow basin). +- LayerNorm: pre-norm strongly preferred (narrow basin). +- Aspect ratio: ~100 (narrow basin). +- n_heads: ~32-128 for typical sizes (wide basin). + +The non-forgiving hyperparameters are the ones where the basin is narrow; these require careful tuning. + +### 5.14 The "messy" nature of architecture design + +The instructor's honest framing: architecture design is **empirical, not theoretical**. There is no closed-form formula for the optimal architecture; instead, people iterate based on: +- Past experiments (Chinchilla, Kaplan, etc.). +- Recent state-of-the-art (LLaMA, Qwen, Gemma, etc.). +- Systems constraints (hardware, parallelization). +- Training stability (avoiding NaN losses). + +This is consistent with the broader campaign theme: **random networks + empirical tuning > theoretical elegance**. + +### 5.15 MoE architecture (deferred to next lecture) + +MoE (Mixture of Experts): each FFN layer is replaced by multiple expert FFNs; a routing network selects a subset of experts per token. + +**Active parameters per token:** reduced (only selected experts are computed). +**Total parameters:** same or higher (more experts = more total parameters). +**Compute:** reduced per token (fewer FFNs per token). +**Performance:** similar or better than dense models at same active parameter count. + +The instructor defers MoE details to the next lecture, but notes that "most new model releases are MoEs." + +### 5.16 Recent variants summary + +| Variant | Year | Key features | +|---|---|---| +| **LLaMA** | 2023 | Pre-norm, RoPE, SwiGLU, no bias, RMSNorm | +| **LLaMA 2** | 2023 | Same as LLaMA, larger training, more data | +| **LLaMA 3** | 2024 | Same architecture, much larger training | +| **Gemma 2** | 2024 | Non-residual post-norm (double norm) | +| **OLMo 2/3** | 2024 | Open-source, non-residual post-norm | +| **Qwen 2/3** | 2024-2025 | Multilingual, larger vocab | +| **GPT-4** | 2023 | Closed; multilingual | +| **Claude** | 2024 | Closed | +| **Gemini** | 2024 | Closed | +| **Percy's Marin** | 2026 | Stanford open foundation model | + +### 5.17 Connections to the campaign + +The architectural choices connect to broader campaign themes: +- **LLaMA template** = a kind of "UFR" in transformer space (per Kumar). +- **Pre-norm + RoPE + SwiGLU** = specific inductive biases (per cs229's geometric deep learning). +- **Forgiving basin** = generic systems: any working parameterization produces interesting behavior (per Fields). +- **Random MoE routing** = a kind of reservoir computing (per brain_counterintuitive). +- **FLOPs as primary control** = scaling laws (per score_dynamics_giorgini's score matching at scale). +- **Architecture vs FLOPs tradeoff** = computational complexity bounds. + +--- + +## 6. Connections + +This section maps the talk's content to the broader 12-video research campaign. + +### 6.1 Backward (cluster A and B foundations) + +#### 6.1.1 `cs229_building_llms_20260621` + +CS229 (the "why" of LLMs) sets the stage for CS336 (the "how"). The Stanford CS336 course explicitly builds on Stanford CS229's content. + +**Connection depth:** Direct. Same course series, different lecture depth. + +#### 6.1.2 `score_dynamics_giorgini_20260621` + +Giorgini's score-based generative modeling is relevant to training dynamics. The Chinchilla/Kaplan scaling laws describe how loss decreases with FLOPs; score matching is one approach to fitting the loss landscape. + +**Connection depth:** Methodological. Both deal with training large generative models. + +#### 6.1.3 `platonic_intelligence_kumar_20260621` + +Kumar's FER vs UFR distinction can be applied to transformer architectures: +- The LLaMA template is a kind of "UFR" — well-engineered structure. +- Random weight initialization is "FER" — entangled, unstructured. +- Training finds a "UFR-like" configuration in the weight space. + +**Connection depth:** Conceptual. Architectural templates as factored representations. + +### 6.2 Forward (cluster E applications) + +#### 6.2.1 `creikey_dl_cv_20260621` + +DDPM (Denoising Diffusion Probabilistic Models) uses transformer-like architectures (U-Net, attention). The CS336 lecture's architectural decisions (pre-norm, RoPE, SwiGLU) apply to DDPM too. + +**Connection depth:** Applied. Same architectural decisions apply. + +### 6.3 Lateral (cluster C connections) + +#### 6.3.1 `brain_counterintuitive_20260621` + +Reservoir computing: fixed random recurrent network + linear readout. Modern transformer architectures have: +- **Pre-norm + RoPE + SwiGLU:** structured inductive biases. +- **MoE routing:** a kind of random projection (each token routed to a subset of experts). + +The "forgiving basin" insight connects to reservoir computing: any working parameterization produces interesting behavior. + +**Connection depth:** Methodological. Random projections vs engineered architectures. + +#### 6.3.2 `generic_systems_fields_20260621` + +Fields' generic systems: any working parameterization produces interesting behavior. The CS336 lecture's "forgiving basin" matches: any of many architectural choices produces a working LLM. + +**Connection depth:** Conceptual. Both emphasize robustness of design choices. + +#### 6.3.3 `entropy_epiplexity_20260621` + +Epiplexity: observer-relative complexity. Different observers (researchers vs students vs practitioners) see different complexities in the same architecture. The "everything you didn't want to know" framing is an epiplexity admission. + +**Connection depth:** Conceptual. + +### 6.4 Cross-cutting themes + +Four themes recur across the campaign and connect to the CS336 lecture: + +1. **Empirical over theoretical** (CS336 + brain_counterintuitive): the field works by experiment, not elegant theory. +2. **Scaling laws** (CS336 + Kaplan + Chinchilla): FLOPs dominate architecture. +3. **Random projections** (CS336 + MoE + reservoir computing): random structure is computationally powerful. +4. **Inductive biases** (CS336 + cs229 geometric deep learning): architecture matters for sample efficiency. + +--- + +## 7. Open Questions + +Sixteen questions arising from this talk that Pass 2 should address. + +### 7.1 Theoretical + +1. **Why are some hyperparameters forgiving and others not?** What makes aspect ratio non-forgiving while head dim is forgiving? + +2. **What is the optimal vocabulary size?** Is there a closed-form formula, or is it always empirical? + +3. **What is the optimal architecture at fixed FLOPs?** ETA's claim is that FLOPs dominate. Is this robust across model scales? + +4. **Why does QK-norm help?** Is the bounded attention score the only reason, or are there deeper effects? + +5. **Why is pre-norm better for stability?** Is this a fundamental property of residual networks, or specific to transformers? + +### 7.2 Empirical + +6. **How does MoE scale vs dense?** Is MoE asymptotically better at fixed FLOPs? + +7. **What is the next architectural breakthrough?** The instructor surveys 2025-2026 models but can't predict the winner. + +8. **Will MoE dominate?** Or will a new paradigm emerge? + +9. **How does hybrid attention (Jamba) compare to pure attention?** Per the lecture: hybrid attention is "one of the trends." + +10. **What is the role of non-residual post-norm (Gemma 2, Olmo 2)?** Is it a fundamental improvement or a workaround for stability issues? + +### 7.3 Applied + +11. **What architecture should I use for my LLM?** Per the instructor: "any LLaMA-like architecture is a safe bet." + +12. **How do I train stably?** Pre-norm + no warmup + QK-norm + gradient clipping. + +13. **How do I scale my LLM?** Fix aspect ratio at ~100, scale d_model and n_layers proportionally. + +14. **Should I use MoE?** Per the instructor: yes, but the details are for the next lecture. + +### 7.4 Philosophical + +15. **Is architecture design science or art?** Per the instructor: "a little bit messy and a little bit complex" — not elegant science. + +16. **Will the field ever converge on a single architecture?** Or will architecture remain in flux? + +--- + +## 8. References + +People, papers, and concepts referenced in the talk and developed in the report. + +### 8.1 People + +| Person | Role | +|---|---| +| Tatsu Hashimoto | Speaker; CS336 co-instructor | +| Percy Liang | CS336 co-instructor (referenced multiple times) | +| Vaswani et al. | Original Transformer paper authors (2017) | +| Touvron et al. | LLaMA authors (2023) | +| Xiong et al. | Pre-LN vs Post-LN analysis (2020) | +| Salazar and Nguyen | BERT analysis (2019) | +| Su et al. | RoPE authors (2021) | +| Shazeer | SwiGLU author (2020) | +| Dehghani et al. | QK-norm authors (2023, Cohere) | +| Kaplan et al. | Scaling laws (2020) | +| Hoffmann et al. | Chinchilla scaling laws (2022) | +| ETA and others | Architecture variation studies (referenced) | + +### 8.2 Papers cited in the talk + +- **Vaswani et al. (2017).** Attention is all you need. *NeurIPS.* +- **Touvron et al. (2023).** LLaMA: open and efficient foundation language models. *arXiv:2302.13971.* +- **Su et al. (2021).** RoFormer: enhanced transformer with rotary position embedding. *arXiv:2104.09864.* +- **Shazeer (2020).** GLU variants improve transformer. *arXiv:2002.05202.* +- **Xiong et al. (2020).** On layer normalization in the transformer architecture. *ICML 2020.* +- **Salazar and Nguyen (2019).** BERT analysis. +- **Dehghani et al. (2023).** Scaling vision transformers to 22 billion parameters. *ICML 2023.* +- **Kaplan et al. (2020).** Scaling laws for neural language models. *arXiv:2001.08361.* +- **Hoffmann et al. (2022).** Training compute-optimal large language models (Chinchilla). *arXiv:2203.15556.* +- **Zhang et al. (2022).** OPT: open pre-trained transformer language models. *arXiv:2205.01068.* +- **Scao et al. (2022).** BLOOM: a 176B-parameter open-access multilingual language model. *arXiv:2211.05100.* +- **Workshop et al. (2022).** BigScience: a case study in the social construction of a multilingual large language model. *arXiv:2212.04960.* +- **Gemma team (2024).** Gemma 2: improving open language models at a practical size. *arXiv:2408.00118.* + +### 8.3 Background references + +- **Bahdanau et al. (2015).** Neural machine translation by jointly learning to align and translate. *ICLR.* +- **Sutskever et al. (2014).** Sequence to sequence learning with neural networks. *NeurIPS.* +- **Hochreiter et al. (2001).** Long short-term memory. *Neural Computation.* +- **He et al. (2016).** Deep residual learning. *CVPR.* +- **Ba et al. (2016).** Layer normalization. *arXiv:1607.06450.* +- **Ioffe & Szegedy (2015).** Batch normalization. *ICML.* + +### 8.4 Internal cross-references + +- **umbrella spec.md** — `conductor/tracks/video_analysis_campaign_20260621/spec.md` — the FR6 8-section report structure. +- **umbrella README.md** — `conductor/tracks/video_analysis_campaign_20260621/README.md` — research-pass framing. +- **child #1 cs229_building_llms** — `conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md` — direct backward; LLM context. +- **child #4 score_dynamics_giorgini** — `conductor/tracks/video_analysis_score_dynamics_giorgini_20260621/report.md` — training dynamics. +- **child #5 platonic_intelligence_kumar** — `conductor/tracks/video_analysis_platonic_intelligence_kumar_20260621/report.md` — representations inside architectures. +- **child #8 brain_counterintuitive** — `conductor/tracks/video_analysis_brain_counterintuitive_20260621/report.md` — reservoir + transformer architectures. +- **child #7 generic_systems_fields** — `conductor/tracks/video_analysis_generic_systems_fields_20260621/report.md` — generic systems + forgiving basin. +- **child #9 neural_dynamics_miller** — `conductor/tracks/video_analysis_neural_dynamics_miller_20260621/report.md` — global control signals vs attention. +- **child #10 multiscale_hoffman** — `conductor/tracks/video_analysis_multiscale_hoffman_20260621/report.md` — Transformers as policies. +- **child #12 creikey_dl_cv** (planned) — DDPM architecture. + +--- + +## Appendix A — Concept Map + +Twenty concepts organized by dependency layer. + +**Layer 0 (the survey lens):** +- Modern transformer census +- LLaMA dominance +- Architecture tradeoffs (generalization, hardware, stability) + +**Layer 1 (the modern block):** +- Pre-norm LayerNorm +- Multi-head self-attention +- RoPE (rotary position embeddings) +- SwiGLU FFN +- Residual connections + +**Layer 2 (variations):** +- Pre-norm vs Post-norm +- QK-norm +- Double norm / non-residual post-norm (Grok, Gemma 2, Olmo 2) +- Hybrid attention (Jamba) +- RMSNorm (vs LayerNorm) + +**Layer 3 (hyperparameters):** +- Vocabulary size (32K monolingual, 256K multilingual) +- Aspect ratio (~100 d_model/n_layers) +- Head dimension (~1 d_model/n_heads) +- n_layers, d_model, d_ff (multiply-by-4 rule) + +**Layer 4 (scaling laws):** +- Kaplan scaling laws (loss ~ FLOPs^-α) +- Chinchilla scaling (compute-optimal models) +- FLOPs as primary control +- Aspect ratio invariance under scale + +**Layer 5 (stability tricks):** +- Pre-norm (better gradient flow) +- No warmup +- QK-norm (bounded attention scores) +- FixNorm (reset optimizer state) +- ScaleNorm (gradient scaling) + +**Layer 6 (recent variants):** +- MoE (deferred to next lecture) +- Gemma 2, Olmo 2 (double norm) +- Grok (double norm) +- Qwen 2/3 (multilingual) +- Percy's Marin (Stanford open model) + +**Layer 7 (connections to campaign):** +- Random projections + reservoir computing +- Forgiving basin + generic systems +- Empirical > theoretical (Levin's mess-as-feature) +- FLOPs dominance + scaling laws + +--- + +## Appendix B — Transcript Excerpts (verbatim, by section) + +### B.1 Opening + +> "So today we're going to talk about architecture, which at least to me has always been pretty inscrutable. Um and so I'm going to take the approach of just telling you kind of everything, right? I'm going to go through all of the modern papers." + +### B.2 Vanilla vs modern + +> "I think many of you have taken an NLP course of some kind or at least seen a transformer, so you've probably seen, you know, the very vanilla transformer from Vaswani et al. [...] we're going to ask you to move the layer norm to the front of each transformer block or the non-residual layers. We're going to ask you to implement something called rope. Um and we're going to ask you to implement something called SwiGLU and not ReLU." + +### B.3 The census motivation + +> "I made this like little table. We'll come back to this little table at the end of the lecture. Um but basically at this point, you know, starting with, you know, the original transformer, there's been actually quite a few autoregressive language models kind of trained on the same class of things." + +### B.4 Dense vs MoE + +> "Um but it turns out if you start looking, there's a lot of different models. Um and so the fact that we have so many different models, most of these actually are MoEs, mixtures of experts, and I'll be talking about that tomorrow rather than today." + +### B.5 Tradeoffs + +> "One of the things that, you know, higher level I want to sort of impress upon you is that architectures are actually a a very complex set of tradeoffs, right? Like what does a architecture have to do? Well, it has to learn from data, so it has to generalize. It has to train efficiently on GPUs. And it has to not blow up, right?" + +### B.6 LLaMA dominance + +> "The dominance of LLaMA-like architectures. Trends over the years (QK-norm, Hybrid attention)." + +### B.7 Pre-LN gradient analysis + +> "Pre-vs-post norm, explanations? Gradient attenuation [Xiong 2020] [...] PreNorm+ScaleNorm+FixNorm+NoWarmup [...] Today - stability and larger LRS for large networks." + +### B.8 Double norm + +> "New things - 'double' norm or non-residual postnorm. If putting LayerNorms in residual streams is bad.. Why not post-norm outside the stream?" + +### B.9 Head dimension sweet spot + +> "I think the thing about head dimensions uh that I'll that I'll end with here is I think this is yet another kind of forgiving hyperparameter. Um there's a couple of ablations that people have done. There's once again a pretty wide basin around one that you can sort of get away with." + +### B.10 Aspect ratio + +> "I think maybe one of the most critical and interesting ones, I think conceptually, is this idea of an aspect ratio, right?" + +### B.11 Wide vs deep parallelization + +> "Width is much easier to parallelize. If you have a really wide model, you know, you can cut that up very easily in your GPUs. Uh tensor parallel is what it's called is much much simpler to deal with." + +### B.12 FLOPs as primary control + +> "I think one of the really interesting things about um transformer hyperparameters is there are a lot of hyperparameters that seem quite important, but they're also fairly forgiving, and people have converged roughly on the minimum. [...] As you increase the FLOPs, the models get better, and that's really controlling the majority of the effects." + +### B.13 ETA result + +> "ETA and others uh did a number of really interesting sort of like architecture um architecture variation experiments, in which their general conclusion on this was that [...] the only thing that matters in some sense is FLOPs." + +### B.14 Vocabulary sizes + +> "I think in the early days of a lot of, you know, um early days of open-source model training, um there were a lot of monolingual models whose only goal was to be good on English. And for those models, you had these like much smaller vocab size, in the 30,000 range." + +### B.15 Multilingual vocab + +> "Post-LLaMA, a lot of people were really interested in multilingual or like production systems. So, these include closed-source models like GPT-4. Um all these have much much larger vocab sizes, and these are roughly in the hundred to 200,000 um vocab range." + +### B.16 Percy's 8B model + +> "And of course I have to give a shout-out to Percy's own 8B model trained with Marine." + +--- + +## Appendix C — Formalizations (expanded) + +### C.1 Transformer block in detail + +The pre-norm LLaMA-like block: + +``` +def transformer_block(x, mask): + # x: (batch, seq_len, d_model) + + # First sub-block: attention + x_norm = rms_norm(x) + q = x_norm @ W_Q # (batch, n_heads, seq_len, head_dim) + k = x_norm @ W_K + v = x_norm @ W_V + + # Apply RoPE + q = apply_rope(q, positions) + k = apply_rope(k, positions) + + # Optional QK-norm + q = layer_norm(q) + k = layer_norm(k) + + # Attention + scores = q @ k.transpose(-1, -2) / sqrt(head_dim) + scores = scores + mask + attn_weights = softmax(scores) + attn_out = attn_weights @ v # (batch, n_heads, seq_len, head_dim) + attn_out = attn_out.transpose(1, 2).reshape(batch, seq_len, d_model) + attn_out = attn_out @ W_O + + # Residual + x = x + attn_out + + # Second sub-block: FFN + x_norm = rms_norm(x) + ffn_out = swiglu(x_norm, W_1, W_2, W_3) + + # Residual + x = x + ffn_out + + return x +``` + +### C.2 RoPE derivation + +For a query q at position p_q and key k at position p_k, RoPE rotates them: + +q' = R(p_q) · q +k' = R(p_k) · k + +The attention score: + +q'^T · k' = (R(p_q) · q)^T · (R(p_k) · k) + = q^T · R(p_q)^T · R(p_k) · k + = q^T · R(p_k - p_q) · k (if R is a rotation group) + +The result depends only on the relative position p_k - p_q. This is the **rotation equivariance** of RoPE. + +For 2D rotation with frequency θ_i: +R(p, θ_i) = [[cos(p · θ_i), -sin(p · θ_i)], [sin(p · θ_i), cos(p · θ_i)]] + +The frequencies θ_i are chosen to cover a range of scales (e.g., θ_i = 10000^(-2i/d) for i = 0, ..., d/2-1). + +### C.3 Pre-norm vs Post-norm gradient analysis + +**Pre-norm block:** y = x + f(LayerNorm(x)) + +∂y/∂x = I + ∂f/∂(LayerNorm(x)) · ∂LayerNorm/∂x + +For a chain of L pre-norm blocks: + +∂L/∂x_0 = ∏_{l=1}^L (I + ∂f_l/∂x_l · ∂LayerNorm/∂x_l) + +The identity I dominates, so the gradient norm is bounded by: + +‖∂L/∂x_0‖ ≤ ‖∂L/∂x_L‖ · ∏_{l=1}^L (1 + ‖∂f_l/∂x_l · ∂LayerNorm/∂x_l‖) + +Even if each term grows exponentially, the product is bounded by the geometric mean of (1 + ‖·‖) terms. For reasonable activations, ‖·‖ ≤ 1, so the gradient doesn't explode. + +**Post-norm block:** y = LayerNorm(x + f(x)) + +∂y/∂x = ∂LayerNorm/∂(x + f(x)) · (I + ∂f/∂x) + +The LayerNorm's Jacobian is ‖∂LayerNorm/∂u‖ ≤ ‖γ‖ / σ_u (where σ_u is the standard deviation of u). If σ_u is small, the LayerNorm amplifies the gradient by ‖γ‖ / σ_u, which can be large. + +This is why post-norm requires warmup: the early training has unstable gradients, and warmup keeps the learning rate low until the gradients stabilize. + +### C.4 QK-norm gradient stability + +Standard attention: Attention(Q, K, V) = softmax(Q · K^T / sqrt(d_k) + mask) · V. + +The attention scores are s = Q · K^T / sqrt(d_k). Without normalization, ‖s‖ can grow unboundedly during training (especially in deep models with high LR). + +QK-norm bounds ‖Q'‖, ‖K'‖ ≤ 1, so ‖s'‖ = ‖Q' · K'^T‖ / sqrt(d_k) ≤ ‖Q'‖ · ‖K'‖ / sqrt(d_k) ≤ 1 / sqrt(d_k). + +For d_k = 128, ‖s'‖ ≤ 0.088. The attention weights are bounded, preventing score explosion. + +### C.5 The aspect ratio and FLOPs tradeoff + +For a transformer with n_layers, d_model, d_ff = 4 · d_model, n_heads = d_model/head_dim: + +**Parameters per layer:** 8 · d_model² (attention) + 2 · d_model · d_ff (FFN) = 8 · d_model² + 8 · d_model² = 16 · d_model². But wait: attention has 4 · d_model² (Q, K, V, O) and FFN has 3 · d_model · d_ff = 12 · d_model². Total: 4 + 12 = 16 · d_model² per layer. + +**FLOPs per token per layer:** 2 · parameters = 32 · d_model² FLOPs. + +**Total FLOPs per token:** n_layers · 32 · d_model² + V · d_model (embedding). + +**Aspect ratio A = d_model / n_layers:** +FLOPs ≈ 32 · d_model³ / A + V · d_model + +For fixed FLOPs, smaller d_model (smaller A) gives less compute. But smaller d_model means narrower hidden dim, less expressiveness per layer. + +**Empirically optimal:** A ≈ 100. For d_model = 4096, n_layers ≈ 40. + +### C.6 Vocabulary size and embedding parameters + +**Embedding parameters:** V · d_model. + +For V = 32K and d_model = 4096: 32K · 4096 = 134M. +For V = 128K and d_model = 4096: 128K · 4096 = 524M. +For V = 256K and d_model = 4096: 256K · 4096 = 1.05B. + +**Output parameters (unembedding):** Same as embedding (V · d_model), often tied with embedding. + +**Total vocabulary params:** V · d_model ≈ V · 4096. + +For a 7B model with d_model = 4096, n_layers ≈ 32: +- Model body: 16 · 4096² · 32 ≈ 8.6B params. +- Vocabulary: V · 4096. For V = 32K: 134M (1.5%); for V = 128K: 524M (6%); for V = 256K: 1.05B (12%). + +For multilingual models with large vocab, the vocabulary is a significant fraction of total params. This is why multilingual models tend to be larger (the instructor's empirical observation). + +### C.7 Scaling laws and FLOPs + +**Kaplan scaling (2020):** L(N) = (N_c / N)^α_N with α_N ≈ 0.076. + +**Chinchilla scaling (2022):** L(N, D) = a/N^α + b/D^β + L_∞ with α = β ≈ 0.34. Optimal N and D scale as √C. + +**The combined result:** at fixed FLOPs C = 6ND: +- N_opt ∝ √C +- D_opt ∝ √C +- L_opt ∝ C^(-0.05) (slow power law) + +**Implication:** smaller models trained on more data are compute-optimal. Doubling N requires doubling D to maintain the optimal ratio. + +### C.8 The head dimension sweet spot + +For a transformer with d_model hidden dim and n_heads heads: +head_dim = d_model / n_heads + +**The attention computation per head:** O(seq_len² · head_dim). +**The total attention:** O(seq_len² · d_model). + +For fixed d_model, varying n_heads doesn't change the total compute; it changes how the compute is parallelized. + +**Empirically optimal head_dim:** ~1 (head_dim ≈ d_model / n_heads ≈ 1, e.g., head_dim = 128 for d_model = 4096, n_heads = 32). Smaller head_dim (64) underfits per head; larger (256) overfits per head and wastes compute. + +--- + +## Appendix D — Connections (expanded) + +### D.1 To `cs229_building_llms_20260621` (in detail) + +CS229 is the introductory Stanford NLP course; CS336 is the deep-dive course on language modeling. The two are sequential in Stanford's curriculum. + +**CS229 topics** (per the summary in child #1): +- EBMs and score matching. +- Attention mechanisms. +- Transformer architectures (overview). +- Training dynamics. + +**CS336 topics** (this child): +- Implementation of transformers from scratch. +- Modern architectures (LLaMA, etc.). +- Hyperparameter choices. +- Stability tricks. +- MoE architectures. + +**CS336 = CS229 + implementation details.** Where CS229 introduces concepts, CS336 dives into the engineering tradeoffs. + +### D.2 To `brain_counterintuitive_20260621` (in detail) + +Reservoir computing: fixed random recurrent network + linear readout. Modern transformer architectures have: +- **Pre-norm + RoPE + SwiGLU:** structured inductive biases (not random). +- **MoE routing:** a kind of random projection (each token routed to a subset of experts). +- **Attention mechanism:** a kind of "soft" addressing (not random). + +The connection: **structured inductive biases and random projections both work** because the loss landscape has a wide forgiving basin (per the instructor's empirical claim). + +### D.3 To `generic_systems_fields_20260621` (in detail) + +Fields' generic systems: any working parameterization produces interesting behavior. The CS336 lecture's "forgiving basin" is a specific empirical observation of this principle. + +**Connection depth:** Conceptual. The forgiving basin in architecture design = generic systems in dynamical systems theory. + +### D.4 To `platonic_intelligence_kumar_20260621` (in detail) + +Kumar's FER vs UFR distinction applied to transformer architectures: +- **Random initialization:** FER (entangled, unstructured). +- **LLaMA template:** UFR (factored: separate components for attention, FFN, LayerNorm). +- **Trained model:** partially UFR (some axes interpretable, others not). + +**Connection depth:** Conceptual. Architectural templates as factored representations. + +### D.5 To `score_dynamics_giorgini_20260621` (in detail) + +Giorgini's score-based generative modeling is one approach to training generative models. The CS336 lecture covers architectures; the score-matching framework provides a training objective. + +**Connection depth:** Methodological. Both deal with generative model training. + +### D.6 To `multiscale_hoffman_20260621` (in detail) + +Hoffman & Prakash's trace logic: Transformers as policies. A Transformer is a Markov matrix on tokens; the policy is the transition probability. + +**Connection depth:** Mathematical. Transformers = specific Markov matrices on trace logic. + +### D.7 To `neural_dynamics_miller_20260621` (in detail) + +Miller's electric field oscillations as global control signals. Transformers use attention as a global control signal (every position can attend to every other position). + +**Connection depth:** Methodological. Global control signals in biological vs artificial systems. + +### D.8 To `creikey_dl_cv_20260621` (planned) + +DDPM as a specific architecture. The CS336 lecture's architectural decisions apply to DDPM U-Nets too. + +**Connection depth:** Applied. Same architectural principles. + +--- + +## Appendix E — Open Questions (expanded) + +### E.1 Theoretical questions + +**E.1.1 Why are some hyperparameters forgiving and others not?** What makes aspect ratio non-forgiving while head dim is forgiving? Is this related to the underlying loss landscape structure? + +**E.1.2 What is the optimal vocabulary size?** Is there a closed-form formula, or is it always empirical? Does it depend on the language distribution? + +**E.1.3 What is the optimal architecture at fixed FLOPs?** ETA's claim is that FLOPs dominate. Is this robust across model scales (1B, 10B, 100B+)? + +**E.1.4 Why does QK-norm help?** Is the bounded attention score the only reason, or are there deeper effects on training dynamics? + +**E.1.5 Why is pre-norm better for stability?** Is this a fundamental property of residual networks (skip connections preserve gradients), or specific to transformers? + +### E.2 Empirical questions + +**E.2.1 How does MoE scale vs dense?** Is MoE asymptotically better at fixed FLOPs? What is the MoE equivalent of Chinchilla? + +**E.2.2 What is the next architectural breakthrough?** The instructor surveys 2025-2026 models but can't predict the winner. Will it be MoE? Hybrid attention? Something new? + +**E.2.3 Will MoE dominate?** Or will a new paradigm emerge? + +**E.2.4 How does hybrid attention (Jamba) compare to pure attention?** Per the lecture: hybrid attention is "one of the trends" but no empirical comparison was given. + +**E.2.5 What is the role of non-residual post-norm (Gemma 2, Olmo 2)?** Is it a fundamental improvement or a workaround for stability issues in pre-norm? + +### E.3 Applied questions + +**E.3.1 What architecture should I use for my LLM?** Per the instructor: "any LLaMA-like architecture is a safe bet." + +**E.3.2 How do I train stably?** Pre-norm + no warmup + QK-norm + gradient clipping. + +**E.3.3 How do I scale my LLM?** Fix aspect ratio at ~100, scale d_model and n_layers proportionally. + +**E.3.4 Should I use MoE?** Per the instructor: yes, but the details are for the next lecture. + +### E.4 Philosophical questions + +**E.4.1 Is architecture design science or art?** Per the instructor: "a little bit messy and a little bit complex" — not elegant science. + +**E.4.2 Will the field ever converge on a single architecture?** Or will architecture remain in flux as new findings emerge? + +--- + +## Appendix F — References (full bibliography) + +### F.1 Primary works cited + +1. Vaswani, A., et al. (2017). Attention is all you need. *NeurIPS 2017.* +2. Touvron, H., et al. (2023). LLaMA: open and efficient foundation language models. *arXiv:2302.13971.* +3. Su, J., et al. (2021). RoFormer: enhanced transformer with rotary position embedding. *arXiv:2104.09864.* +4. Shazeer, N. (2020). GLU variants improve transformer. *arXiv:2002.05202.* +5. Xiong, R., et al. (2020). On layer normalization in the transformer architecture. *ICML 2020.* +6. Dehghani, M., et al. (2023). Scaling vision transformers to 22 billion parameters. *ICML 2023.* +7. Kaplan, J., et al. (2020). Scaling laws for neural language models. *arXiv:2001.08361.* +8. Hoffmann, J., et al. (2022). Training compute-optimal large language models (Chinchilla). *arXiv:2203.15556.* + +### F.2 Foundational references + +9. Bahdanau, D., et al. (2015). Neural machine translation by jointly learning to align and translate. *ICLR 2015.* +10. Sutskever, I., et al. (2014). Sequence to sequence learning with neural networks. *NeurIPS 2014.* +11. Hochreiter, S., et al. (2001). Long short-term memory. *Neural Computation*, 9(8), 1735-1780. +12. He, K., et al. (2016). Deep residual learning for image recognition. *CVPR 2016.* +13. Ba, J. L., et al. (2016). Layer normalization. *arXiv:1607.06450.* +14. Ioffe, S., & Szegedy, C. (2015). Batch normalization. *ICML 2015.* + +### F.3 Recent model papers + +15. Scao, T. L., et al. (2022). BLOOM: a 176B-parameter open-access multilingual language model. *arXiv:2211.05100.* +16. Zhang, S., et al. (2022). OPT: open pre-trained transformer language models. *arXiv:2205.01068.* +17. Workshop, B., et al. (2022). BigScience: a case study in the social construction of a multilingual large language model. *arXiv:2212.04960.* +18. Touvron, H., et al. (2023). LLaMA 2: open foundation and fine-tuned chat models. *arXiv:2307.09288.* +19. AI@Meta (2024). Llama 3 model card. +20. Gemma Team (2024). Gemma 2: improving open language models at a practical size. *arXiv:2408.00118.* +21. OLMo Team (2024). OLMo 2: 2.5B and 32B language models. +22. Qwen Team (2024). Qwen 2 technical report. +23. Bai, J., et al. (2024). Qwen3 technical report. *arXiv:2509.17177.* + +### F.4 Architecture pattern papers + +24. Press, O., et al. (2022). ALiBi: attention with linear biases. +25. Su, J., et al. (2024). RoPE: rotary position embedding. +26. Bricken, T., et al. (2023). Monosemanticity in transformer language models (Anthropic). +27. Park, J., et al. (2024). Jamba: hybrid transformer-mamba language model. + +--- + +## Appendix G — Cross-references within campaign + +### G.1 Backward references + +- **cs229_building_llms_20260621** (§6.1.1): direct backward; LLM context. +- **score_dynamics_giorgini_20260621** (§6.1.2): training dynamics. +- **platonic_intelligence_kumar_20260621** (§6.1.3): representations inside architectures. +- **brain_counterintuitive_20260621** (§6.3.1): reservoir + transformer architectures. +- **generic_systems_fields_20260621** (§6.3.2): generic systems + forgiving basin. +- **neural_dynamics_miller_20260621** (§6.3.4): global control signals. +- **multiscale_hoffman_20260621** (§6.3.3): Transformers as policies. + +### G.2 Forward references + +- **creikey_dl_cv_20260621** (planned): DDPM architecture. + +### G.3 Reference dependency graph + +``` +Stanford CS229 (LLM basics) + | + v +Stanford CS336 Lecture 3: Architectures + | + +----> LLaMA template (pre-norm + RoPE + SwiGLU + RMSNorm) + | | + | +----> QK-norm + | +----> Double norm (Gemma 2, Olmo 2) + | +----> MoE (next lecture) + | + +----> Hyperparameter sweet spots + | | + | +----> Vocabulary (32K-256K) + | +----> Aspect ratio (~100) + | +----> Head dim (~1) + | + +----> Stability tricks + | | + | +----> No warmup + | +----> QK-norm + | +----> FixNorm / ScaleNorm + | + +----> Scaling laws + | | + | +----> Kaplan (2020) + | +----> Chinchilla (2022) + | + +----> Recent models + | + +----> LLaMA 2/3 (Meta) + +----> Gemma 2/3 (Google) + +----> OLMo 2/3 (Allen AI) + +----> Qwen 2/3 (Alibaba) + +----> GPT-4, Claude, Gemini (closed) + +----> Percy's Marin (Stanford) +``` + +--- + +## Appendix H — Synthesis Summary + +A single-paragraph TL;DR of the talk, suitable for a busy reader. + +Stanford CS336 Lecture 3: Architectures is a deep technical survey of modern transformer architecture decisions in language modeling. The instructor (Tatsu Hashimoto) walks through the LLaMA template (pre-norm LayerNorm + RoPE + SwiGLU FFN + RMSNorm + no bias) that has become the de facto standard for open-source dense LLMs, then surveys common variations (QK-norm for attention stability, double-norm / non-residual post-norm in Gemma 2 and Olmo 2, hybrid attention in Jamba). Most architectural hyperparameters are forgiving — there's a wide basin of good values for vocabulary size (32K monolingual to 256K multilingual), head dimension (~1 d_model/n_heads), and most other choices. The non-forgiving hyperparameters are aspect ratio (~100 d_model/n_layers, driven by parallelization constraints) and activation (SwiGLU preferred over ReLU). Training stability tricks (no warmup for pre-norm, QK-norm, FixNorm, ScaleNorm) are architecture-coupled. Per Chinchilla/Kaplan scaling laws, FLOPs dominate architecture — at fixed compute, smaller models trained on more data outperform larger models trained on less. The instructor honestly frames architecture design as empirical, messy, and driven by "learning from others' experience" rather than elegant theory — a stance consistent with the broader campaign theme of empirical > theoretical. Most new model releases in 2025-2026 are MoE (deferred to next lecture). The lecture closes by connecting back to CS229 and the A1 implementation assignment (move LayerNorm to front, implement RoPE, implement SwiGLU). + +--- + +## Appendix I — Personal Notes + +Things to revisit in Pass 2 (the user's de-obfuscation pass). + +1. **The "forgiving basin" claim** deserves deeper investigation. Is there a theoretical explanation for why some hyperparameters are forgiving? Pass 2 should explore the loss landscape structure and its relation to hyperparameter sensitivity. + +2. **The aspect ratio = ~100 sweet spot** is a concrete empirical claim that could be formalized. Pass 2 should specify the optimization problem and the relationship to parallelization constraints. + +3. **The pre-norm vs post-norm gradient analysis** (§5.4) is a clean mathematical result. Pass 2 should formalize the gradient norm bounds and explore whether post-norm can be saved by better initialization. + +4. **The connection to MoE** is mentioned but not detailed. Pass 2 should explore how MoE routing relates to reservoir computing (random projections). + +5. **The Percy Liang / Marin connection** is a current Stanford project. Pass 2 should explore what the Marin model architecture is and how it compares to the LLaMA template. + +6. **The QK-norm vs FixNorm comparison** is implicit. Both are stability tricks; QK-norm changes the architecture, FixNorm changes the optimizer. Pass 2 should compare. + +7. **The "FLOPs dominate" claim** is the strongest theoretical result. Pass 2 should explore whether this is consistent with the Chinchilla scaling laws and whether architecture-specific differences exist. + +8. **The non-residual post-norm (Gemma 2, Olmo 2)** is a recent trend. Pass 2 should investigate why pre-norm's gradient flow argument doesn't apply, and whether non-residual post-norm is fundamentally different. + +9. **The hybrid attention (Jamba)** is mentioned. Pass 2 should explore how mixing attention with SSMs affects the trace logic framework. + +10. **The relationship to Fields' generic systems** could be deepened: the "forgiving basin" is a specific instantiation of "any working parameterization produces interesting behavior." + +--- + +## Appendix J — Glossary + +| Term | Definition | +|---|---| +| **Pre-norm** | LayerNorm applied before the sublayer in a transformer block. | +| **Post-norm** | LayerNorm applied after the sublayer (added to residual). Original Transformer (Vaswani 2017). | +| **QK-norm** | LayerNorm applied to query and key vectors before the dot product. | +| **RoPE** | Rotary Position Embedding; encodes position via rotation of Q and K. | +| **RMSNorm** | Root Mean Square LayerNorm; no mean-centering. | +| **SwiGLU** | Gated Linear Unit with SiLU activation; preferred FFN in modern LLMs. | +| **MoE** | Mixture of Experts; FFN replaced by multiple experts + router. | +| **Aspect ratio** | d_model / n_layers; ~100 for modern LLMs. | +| **Head dimension** | d_model / n_heads; ~1 for modern LLMs. | +| **Vocabulary size** | V (tokenizer output space); 32K-256K depending on language coverage. | +| **BPE** | Byte Pair Encoding; subword tokenization. | +| **SentencePiece** | Subword tokenization library. | +| **Kaplan scaling** | Power laws for loss vs N, D, C. | +| **Chinchilla scaling** | Compute-optimal models: N ∝ √C, D ∝ √C. | +| **FLOPs** | Floating-point operations; the dominant variable for compute. | +| **Pipeline parallelism** | Splitting layers across GPUs (slow due to sequential dependencies). | +| **Tensor parallelism** | Splitting neurons across GPUs (fast). | +| **Forgiving basin** | Wide range of hyperparameter values that produce good models. | +| **Double norm** | Two LayerNorms per transformer block (Gemma 2, Olmo 2). | +| **Non-residual post-norm** | Double norm outside the residual stream. | +| **Hybrid attention** | Mixing self-attention with SSM (state-space model) layers. | +| **FixNorm** | Reset optimizer state after training instabilities. | +| **ScaleNorm** | Scale gradients by 1/sqrt(depth). | +| **Tatsu Hashimoto** | CS336 co-instructor (the speaker). | +| **Percy Liang** | CS336 co-instructor (referenced). | +| **Marin** | Stanford open foundation model project (Percy Liang's). | +| **A1 assignment** | CS336 assignment 1: implement a transformer from scratch. | + +--- + +*End of report. Lossless preservation per umbrella spec §0. Pass 2 (de-obfuscation) and Pass 3 (projection to applied domain) to follow.* diff --git a/conductor/tracks/video_analysis_cs336_architectures_20260621/summary.md b/conductor/tracks/video_analysis_cs336_architectures_20260621/summary.md new file mode 100644 index 00000000..a0bcba83 --- /dev/null +++ b/conductor/tracks/video_analysis_cs336_architectures_20260621/summary.md @@ -0,0 +1,25 @@ +# Summary: Stanford CS336 Lecture 3: Architectures + +**Source:** https://youtu.be/lVynu4bo1rY +**Author:** Stanford CS336 Spring 2026 (Tatsu Hashimoto) +**Track:** Child #11 of `video_analysis_campaign_20260621` +**Cluster:** E (Stanford course VODs >1hr) +**Pass:** 1 of 3 (research-only deep-dive) + +--- + +## One-paragraph synthesis + +Stanford CS336 Lecture 3: Architectures is a deep technical survey of modern transformer architecture decisions in language modeling. The instructor (Tatsu Hashimoto) walks through the LLaMA template (pre-norm LayerNorm + RoPE + SwiGLU FFN + RMSNorm + no bias) that has become the de facto standard for open-source dense LLMs, then surveys common variations (QK-norm for attention stability, double-norm / non-residual post-norm in Gemma 2 and Olmo 2, hybrid attention in Jamba). Most architectural hyperparameters are forgiving — there's a wide basin of good values for vocabulary size (32K monolingual to 256K multilingual), head dimension (~1 d_model/n_heads), and most other choices. The non-forgiving hyperparameters are aspect ratio (~100 d_model/n_layers, driven by parallelization constraints) and activation (SwiGLU preferred over ReLU). Training stability tricks (no warmup for pre-norm, QK-norm, FixNorm, ScaleNorm) are architecture-coupled. Per Chinchilla/Kaplan scaling laws, FLOPs dominate architecture — at fixed compute, smaller models trained on more data outperform larger models trained on less. The instructor honestly frames architecture design as empirical, messy, and driven by "learning from others' experience" rather than elegant theory. **Backward connections:** cs229_building_llms (direct; LLM context), platonic_intelligence_kumar (architectures as UFR), brain_counterintuitive (reservoir + transformer), generic_systems_fields (forgiving basin), score_dynamics_giorgini (training dynamics), neural_dynamics_miller (global control signals), multiscale_hoffman (Transformers as policies). **Forward connections:** creikey_dl_cv (DDPM architecture). + +--- + +## Three key takeaways + +1. **The LLaMA template is the standard** — pre-norm LayerNorm + RoPE + SwiGLU FFN + RMSNorm + no bias. This combination has converged across most open-source dense LLMs (LLaMA 2/3, OLMo, Gemma, Qwen). Architecture design is now "forgiving" within this template's basin. +2. **FLOPs dominate architecture** — at fixed compute, scaling laws predict smaller models trained on more data beat larger models trained on less. Architectural choices are secondary to compute budget. The non-forgiving hyperparameters (aspect ratio ~100, SwiGLU activation) are driven by systems constraints. +3. **Architecture is messy empirical work** — the instructor's honest framing: "everything you didn't want to know about architectures and hyperparameters." No elegant theory; just experience-based conventions. Most modern variants (QK-norm, double norm, hybrid attention) are stability tricks coupled to the LLaMA template. + +--- + +*Pass 2 (de-obfuscation via user's mathematical encoding) to follow.*