conductor(cs336_architectures): Phase 4 Synthesis - report.md (1442 lines, 70KB) + summary.md (~400 words)

conductor/tracks/video_analysis_cs336_architectures_20260621/summary.md

@@ -0,0 +1,25 @@
+# Summary: Stanford CS336 Lecture 3: Architectures
+
+**Source:** https://youtu.be/lVynu4bo1rY
+**Author:** Stanford CS336 Spring 2026 (Tatsu Hashimoto)
+**Track:** Child #11 of `video_analysis_campaign_20260621`
+**Cluster:** E (Stanford course VODs >1hr)
+**Pass:** 1 of 3 (research-only deep-dive)
+
+---
+
+## One-paragraph synthesis
+
+Stanford CS336 Lecture 3: Architectures is a deep technical survey of modern transformer architecture decisions in language modeling. The instructor (Tatsu Hashimoto) walks through the LLaMA template (pre-norm LayerNorm + RoPE + SwiGLU FFN + RMSNorm + no bias) that has become the de facto standard for open-source dense LLMs, then surveys common variations (QK-norm for attention stability, double-norm / non-residual post-norm in Gemma 2 and Olmo 2, hybrid attention in Jamba). Most architectural hyperparameters are forgiving — there's a wide basin of good values for vocabulary size (32K monolingual to 256K multilingual), head dimension (~1 d_model/n_heads), and most other choices. The non-forgiving hyperparameters are aspect ratio (~100 d_model/n_layers, driven by parallelization constraints) and activation (SwiGLU preferred over ReLU). Training stability tricks (no warmup for pre-norm, QK-norm, FixNorm, ScaleNorm) are architecture-coupled. Per Chinchilla/Kaplan scaling laws, FLOPs dominate architecture — at fixed compute, smaller models trained on more data outperform larger models trained on less. The instructor honestly frames architecture design as empirical, messy, and driven by "learning from others' experience" rather than elegant theory. **Backward connections:** cs229_building_llms (direct; LLM context), platonic_intelligence_kumar (architectures as UFR), brain_counterintuitive (reservoir + transformer), generic_systems_fields (forgiving basin), score_dynamics_giorgini (training dynamics), neural_dynamics_miller (global control signals), multiscale_hoffman (Transformers as policies). **Forward connections:** creikey_dl_cv (DDPM architecture).
+
+---
+
+## Three key takeaways
+
+1. **The LLaMA template is the standard** — pre-norm LayerNorm + RoPE + SwiGLU FFN + RMSNorm + no bias. This combination has converged across most open-source dense LLMs (LLaMA 2/3, OLMo, Gemma, Qwen). Architecture design is now "forgiving" within this template's basin.
+2. **FLOPs dominate architecture** — at fixed compute, scaling laws predict smaller models trained on more data beat larger models trained on less. Architectural choices are secondary to compute budget. The non-forgiving hyperparameters (aspect ratio ~100, SwiGLU activation) are driven by systems constraints.
+3. **Architecture is messy empirical work** — the instructor's honest framing: "everything you didn't want to know about architectures and hyperparameters." No elegant theory; just experience-based conventions. Most modern variants (QK-norm, double norm, hybrid attention) are stability tricks coupled to the LLaMA template.
+
+---
+
+*Pass 2 (de-obfuscation via user's mathematical encoding) to follow.*