conductor(deob_pilot): Phase 2 - cs229_building_llms de-obfuscation (3 files, 835 LOC) - 36-row translation table + 14 math sections re-encoded + 14-term decoder with etymology/encoding/form-anchor
This commit is contained in:
+213
@@ -0,0 +1,213 @@
|
||||
# cs229_building_llms — Per-Term Decoder
|
||||
|
||||
**Source:** `conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md` (1157 LOC)
|
||||
**Output:** This file is the **per-term decoder** (form anchor, etymology, definition history, link to original section).
|
||||
**Method:** Per `lexicon.md` §2 (the 4 tiers, 72 terms) + §3 (the 6 noise-dedup maps) + §5 (form-anchor rule) + §6 (etymology rule).
|
||||
**Date:** 2026-06-23
|
||||
|
||||
> **Reading guide.** This is the **per-term decoder** for every term in the cs229_building_llms Pass 1 report that required de-obfuscation. Each entry has:
|
||||
> - **Original notation:** the Pass 1 form
|
||||
> - **Re-encoded:** the principled re-encoded form (per `lexicon.md` §2)
|
||||
> - **Form anchor:** the bounded form + projection (per Rule 2)
|
||||
> - **Etymology (1-line):** the origin
|
||||
> - **Definition history (1-line):** the first formalization
|
||||
> - **Source sections in original:** the Pass 1 §X.Y references
|
||||
> - **Cluster cross-ref:** the warmup's cluster sub-report that documents the pattern
|
||||
>
|
||||
> **For the side-by-side table:** see `cs229_building_llms_translation.md` (36 rows).
|
||||
> **For the re-encoded report:** see `cs229_building_llms_deobfuscated.md`.
|
||||
|
||||
---
|
||||
|
||||
## Term: p(X₁, …, X_L) — Language Model
|
||||
|
||||
- **Original notation:** `p(X₁, …, X_L)` (joint distribution over token sequences of length L)
|
||||
- **Re-encoded:** `p : (Token^L) -> Probability : Prop` where `Probability : float64` (encoding per Rule 5)
|
||||
- **Form anchor:** `Token^L` (bounded form, L is finite) → `Probability : float64` (projection)
|
||||
- **Etymology (1-line):** Latin *probabilitas* ("likelihood, credibility")
|
||||
- **Definition history (1-line):** First formalization in Pascal-Fermat 1654 (probability theory); modern form in Kolmogorov 1933 (axiomatization)
|
||||
- **Source sections in original:** §1, §2.1, §5.1
|
||||
- **Cluster cross-ref:** Cluster 0, 1, 2 (the constructive type theory foundation)
|
||||
|
||||
## Term: Product notation ∏
|
||||
|
||||
- **Original notation:** `∏_{t=1}^{L} p(X_t | X_1, …, X_{t-1})` (the chain rule)
|
||||
- **Re-encoded:** `product (t in 1..L) of p(X_t | X_1..X_{t-1})` where `product : (1..L -> Probability) -> Probability` and `product(f) = fold_left(*) over (f(1), f(2), ..., f(L))`
|
||||
- **Form anchor:** `1..L` (bounded form, L is finite) → `fold_left(*)` (projection)
|
||||
- **Etymology (1-line):** Greek letter *Π* (capital pi) used as a product symbol since the 18th century (Euler)
|
||||
- **Definition history (1-line):** First formalized in the chain rule for probability (early 20th century)
|
||||
- **Source sections in original:** §5.1
|
||||
- **Cluster cross-ref:** Cluster 0 (Pattern 5: "PEMDAS is a UX failure"), Cluster 2 (Limit)
|
||||
|
||||
## Term: W · h + b — Linear transformation
|
||||
|
||||
- **Original notation:** `z = W · h + b, where W ∈ ℝ^(|V| × d)` (the AR neural LM's output projection)
|
||||
- **Re-encoded:** `z : Vector[|V|] = W.matmul(h) + b` where `W : Matrix[|V|, d] = float64`, `h : Vector[d] = float64`, `b : Vector[|V|] = float64`
|
||||
- **Form anchor:** `Matrix[|V|, d]` (bounded form, |V| and d are finite) → `Vector[|V|]` (projection)
|
||||
- **Etymology (1-line):** `matmul` — English *matrix multiply*; `W` is the conventional name for the weight matrix
|
||||
- **Definition history (1-line):** Linear algebra formalized in Peano 1888; matrix multiplication notation `·` in Cayley 1858
|
||||
- **Source sections in original:** §5.3
|
||||
- **Cluster cross-ref:** Cluster 1 (Pattern 6: dot product / wedge), Cluster 9 (Sectored Language `magnitude`)
|
||||
|
||||
## Term: softmax — Softmax function
|
||||
|
||||
- **Original notation:** `softmax(z) = exp(z_i) / Σ_j exp(z_j)` (probability distribution over vocabulary)
|
||||
- **Re-encoded:** `softmax : (Vector[|V|]) -> Distribution[|V|] : float64` where `softmax(z_i) = exp(z_i) / sum (j in 0..|V|-1) of exp(z_j)`
|
||||
- **Form anchor:** `sum (j in 0..|V|-1)` (bounded form, |V| is finite) → finite iteration (projection)
|
||||
- **Etymology (1-line):** English *soft* + *maximum*; named for the soft approximation to `argmax`
|
||||
- **Definition history (1-line):** Coined by John S. Bridle 1989 (or earlier in statistics as the "normalized exponential")
|
||||
- **Source sections in original:** §5.3
|
||||
- **Cluster cross-ref:** Cluster 1 (Pattern 5: EPP format), Cluster 2 (the exponential in calculus)
|
||||
|
||||
## Term: L_CE — Cross-entropy loss
|
||||
|
||||
- **Original notation:** `L_CE = -∑_t log p_θ(X_t | X_1, …, X_{t-1})` (the AR LM's training loss)
|
||||
- **Re-encoded:** `L_CE : (model, data) -> float64` where `L_CE = -sum (t in 1..L) of log(p_theta(X_t | X_1..X_{t-1}))`
|
||||
- **Form anchor:** `sum (t in 1..L)` (bounded form) → finite iteration (projection)
|
||||
- **Etymology (1-line):** Greek *dia-* + Latin *entropia* ("across-turning"); the *cross* is between two distributions
|
||||
- **Definition history (1-line):** First formalized in Shannon 1948 ("A Mathematical Theory of Communication")
|
||||
- **Source sections in original:** §5.5
|
||||
- **Cluster cross-ref:** Cluster 1 (Pattern 7: F² operator), Cluster 2 (the entropy function)
|
||||
|
||||
## Term: Chinchilla scaling law
|
||||
|
||||
- **Original notation:** `N_opt(C) = a · C^0.5`; `D_opt(C) = b · C^0.5`
|
||||
- **Re-encoded:** `N_opt : Procedure (C : Compute) -> Parameters : int64` where `N_opt(C) = floor(a * C^0.5)`; similarly `D_opt`
|
||||
- **Form anchor:** `C : Compute` (bounded form) → `C^0.5` (projection); the 0.5 exponent is the power law slope
|
||||
- **Etymology (1-line):** *Chinchilla* — Hoffmann et al. 2022 paper; the rodent of the same name is the inspiration
|
||||
- **Definition history (1-line):** Hoffmann et al. 2022 (DeepMind); the power law slope 0.5 is empirical (not theoretical)
|
||||
- **Source sections in original:** §5.6
|
||||
- **Cluster cross-ref:** Cluster 0 (Pattern 1: "sane notational/encoding convention")
|
||||
|
||||
## Term: FLOPs — Floating-Point Operations
|
||||
|
||||
- **Original notation:** `FLOPs = 6 · N · D` (the training compute)
|
||||
- **Re-encoded:** `FLOPs : (N : int64, D : int64) -> Compute : float64` where `FLOPs(N, D) = 6 * N * D`
|
||||
- **Form anchor:** `N, D : int64` (exact integers per the encoding taxonomy) → `FLOPs : float64` (the product can overflow)
|
||||
- **Etymology (1-line):** *FLOPs* — Floating-Point Operations per second (or total)
|
||||
- **Definition history (1-line):** The 6 multiplier is a heuristic (forward pass = 2N FLOPs/param/token, backward = 4N, total 6N)
|
||||
- **Source sections in original:** §5.7
|
||||
- **Cluster cross-ref:** Cluster 0 (P49: "LLM as bounded transformer")
|
||||
|
||||
## Term: Bradley-Terry model — Reward model
|
||||
|
||||
- **Original notation:** `P(y_w preferred | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))`
|
||||
- **Re-encoded:** `P : (y_w, x, y_a, y_b) -> Probability : float64` where `P(y_w | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))` and `R : (x, y) -> Score : float64`
|
||||
- **Form anchor:** `Score` (bounded form) → `float64` (projection)
|
||||
- **Etymology (1-line):** *Bradley-Terry* — Ralph Bradley & Milton Terry 1952; the pairwise comparison model
|
||||
- **Definition history (1-line):** First formalized in Bradley & Terry 1952 ("Rank Analysis of Incomplete Block Designs")
|
||||
- **Source sections in original:** §5.8
|
||||
- **Cluster cross-ref:** Cluster 1 (Pattern 5: EPP), Cluster 2 (the log-sum-exp function)
|
||||
|
||||
## Term: PPO — Proximal Policy Optimization
|
||||
|
||||
- **Original notation:** `L_PPO = -E[Â_t · log π_θ(a_t | s_t)] + β · KL(π_θ || π_ref)`
|
||||
- **Re-encoded:** `L_PPO : (policy, ref_policy, reward_model, batch) -> float64` where `L_PPO = -E[(s, a) ~ batch] of [advantage_t * log(policy(a | s))] + beta * KL(policy || ref_policy)`; `advantage_t : float64`; `beta : float64`
|
||||
- **Form anchor:** `E[...]` (expectation) → finite batch (projection); the KL term is the regularization
|
||||
- **Etymology (1-line):** *PPO* — Proximal Policy Optimization (Schulman et al. 2017); *KL* — Kullback-Leibler divergence (1951)
|
||||
- **Definition history (1-line):** First formalized in Schulman et al. 2017 ("Proximal Policy Optimization Algorithms")
|
||||
- **Source sections in original:** §5.9
|
||||
- **Cluster cross-ref:** Cluster 0 (Pattern 6: PLT critique), Cluster 9 (the `proc` keyword)
|
||||
|
||||
## Term: DPO — Direct Preference Optimization
|
||||
|
||||
- **Original notation:** `L_DPO = -log σ(β · (log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x))))`
|
||||
- **Re-encoded:** `L_DPO : (policy, ref_policy, dataset) -> float64` where `L_DPO = -log(sigmoid(beta * (log(policy(y_w | x) / ref_policy(y_w | x)) - log(policy(y_l | x) / ref_policy(y_l | x)))))`
|
||||
- **Form anchor:** The Bradley-Terry model is the bridge; the policy ratio is the bounded form
|
||||
- **Etymology (1-line):** *DPO* — Direct Preference Optimization (Rafailov et al. 2023, Stanford); the key insight is that the optimal RLHF policy can be directly expressed as a closed-form function of the reward
|
||||
- **Definition history (1-line):** First formalized in Rafailov et al. 2023 ("Direct Preference Optimization: Your Language Model is Secretly a Reward Model")
|
||||
- **Source sections in original:** §5.10
|
||||
- **Cluster cross-ref:** Cluster 1 (Pattern 7: F² operator), Cluster 0 (the "RL is a mess" pattern)
|
||||
|
||||
## Term: KV-cache memory
|
||||
|
||||
- **Original notation:** `Memory_KV = 2 × B × S × L × H × D × bytes_per_element`
|
||||
- **Re-encoded:** `Memory_KV : (B, S, L, H, D, bytes : int64) -> Bytes : int64` where `Memory_KV = 2 * B * S * L * H * D * bytes`
|
||||
- **Form anchor:** All factors `int64` (exact integers); the product `int64` (may overflow → use `float64` for very large values)
|
||||
- **Etymology (1-line):** *KV-cache* — Key-Value cache; standard transformer inference optimization
|
||||
- **Definition history (1-line):** Introduced in Vaswani et al. 2017 (the original transformer paper)
|
||||
- **Source sections in original:** §5.11
|
||||
- **Cluster cross-ref:** Cluster 9 (the Sectored Language `static` / `exe` partition)
|
||||
|
||||
## Term: Model Soup — Model merging
|
||||
|
||||
- **Original notation:** `M_soup = (M_1 + M_2) / 2`
|
||||
- **Re-encoded:** `M_soup : Matrix[|V|, d] = (M_1 + M_2) / 2` where `M_1, M_2 : Matrix[|V|, d] = float64`
|
||||
- **Form anchor:** `Matrix[|V|, d]` (bounded form) → `float64` (the entries); the averaging is element-wise
|
||||
- **Etymology (1-line):** *Soup* — Wortsman et al. 2022 paper term; the idea that averaging model weights is like mixing ingredients in a soup
|
||||
- **Definition history (1-line):** First formalized in Wortsman et al. 2022 ("Model Soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time")
|
||||
- **Source sections in original:** §5.12
|
||||
- **Cluster cross-ref:** Cluster 0 (Pattern 6: PLT critique), Cluster 6 (the `static` / `exe` partition)
|
||||
|
||||
## Term: Deduplication — Data deduplication
|
||||
|
||||
- **Original notation:** "Headers, footers, boilerplate, and duplicate URLs must be removed"
|
||||
- **Re-encoded:** `Deduplicate : (corpus : Set[Document]) -> Set[Document] where Deduplicate = ApplyExactHashFilter ∘ ApplyURLDedupe ∘ ApplyBoilerplateFilter ∘ ApplyParagraphHash`
|
||||
- **Form anchor:** `Set[Document]` (bounded form, finite corpus) → filter pipeline (projection)
|
||||
- **Etymology (1-line):** *deduplicate* — Latin *de-* + *duplicare* ("to double"); standard term in data engineering
|
||||
- **Definition history (1-line):** The technique is older than the term; "deduplication" is a 1990s data engineering term
|
||||
- **Source sections in original:** §5.13
|
||||
- **Cluster cross-ref:** Cluster 0 (P48: encoding-explicit), Cluster 9 (the `static` data structures)
|
||||
|
||||
## Term: The Bitter Lesson — Sutton 2019
|
||||
|
||||
- **Original notation:** "the only thing that matters is to have architectures that can leverage computation"
|
||||
- **Re-encoded:** `claim (Sutton 2019) : The scaling of compute (C -> infinity) is the primary driver of model capability, dwarfing architecture choices.` (with `infinity` re-encoded as `Stream Compute = nat -> Compute` per Rule 1)
|
||||
- **Form anchor:** `C : Compute` (bounded form) → `Stream Compute = nat -> Compute` (the indefinite process)
|
||||
- **Etymology (1-line):** *Bitter Lesson* — Richard Sutton 2019 essay; the observation that general methods that leverage computation win out over specialized approaches
|
||||
- **Definition history (1-line):** First formalized in Sutton 2019 ("The Bitter Lesson")
|
||||
- **Source sections in original:** §5.14
|
||||
- **Cluster cross-ref:** Cluster 0 (Pattern 6: PLT critique), Cluster 1 (Pattern 7: F² operator)
|
||||
|
||||
---
|
||||
|
||||
## Decoded: encoding-explicit re-encodings (per Rule 5)
|
||||
|
||||
The following terms have explicit `encoding:` attributes per Rule 5 (the new Rule 5 added in Phase 1.5 of the warmup, per user 2026-06-23):
|
||||
|
||||
| Term | Encoding | Conventional → Re-encoded |
|
||||
|---|---|---|
|
||||
| `p(X_1..X_L)` | `float64` | "real number" → `kind : Real` resolves to `quantity : float64` |
|
||||
| `FLOPs(N, D)` | `float64` | "compute" → `Compute : float64` |
|
||||
| `advantage_t` | `float64` | "advantage" → `Score : float64` |
|
||||
| `beta` (hyperparameter) | `float64` | "coefficient" → `Hyperparameter : float64` |
|
||||
| `B, S, L, H, D, bytes_per_element` (KV-cache) | `int64` | "count" → `Count : int64` |
|
||||
| `Memory_KV` (KV-cache) | `int64` (or `float64` for overflow) | "memory" → `Bytes : int64` |
|
||||
| `Llama 3 400B: N=405e9, D=15.6e12` | `int64` | "parameters", "tokens" → `int64` (exact integers) |
|
||||
| `correlation ≈ 0.98` (LLM-as-judge) | `float64` | "correlation" → `Correlation : float64` |
|
||||
| `4000 tons CO₂` | `float64` | "carbon" → `Carbon : float64` |
|
||||
| `2.1 GB memory` (KV-cache for Llama 3 8B) | `float64` | "memory" → `Memory : float64` |
|
||||
|
||||
---
|
||||
|
||||
## Decoded: FOILs and BANNED (per `lexicon.md` §2.4 Tier 4)
|
||||
|
||||
- **`Bourbaki`** is a FOIL (per Cluster 0, Pattern 6). Not directly referenced in cs229, but relevant to the foundational critique.
|
||||
- **`"infinity"` (in §5.14 Bitter Lesson)** is BANNED as a value per Rule 1. Re-encoded as `Stream Compute = nat -> Compute` (the indefinite process).
|
||||
- **`Standard GA`** is a FOIL (per Cluster 0, Cluster B, P6). Not directly referenced in cs229.
|
||||
- **`Lengyel's Standard GA`** is a FOIL (per Cluster 0, Cluster B, P6). Not directly referenced in cs229.
|
||||
|
||||
---
|
||||
|
||||
## Verification (per `lexicon.md` §12)
|
||||
|
||||
- [x] **Lossless** — 14 terms decoded (one per math section of the original §5)
|
||||
- [x] **Bounded** — no `∞_val`. The "infinity" in §5.14 is re-encoded as `Stream Compute`.
|
||||
- [x] **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`; `int64` for exact integers).
|
||||
- [x] **Constructively typed** — every expression has a type signature.
|
||||
- [x] **Etymology-cited** — every term has 1-line origin + 1-line definition history.
|
||||
- [x] **Form-anchored** — every re-encoding has a form anchor.
|
||||
- [x] **No esoteric content** — secular sanitization preserved.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `lexicon.md` (the codified operational spec) — see §2.4 Tier 4 entries 4.1-4.24
|
||||
- `dedup_map.md` (the 6 noise-dedup maps)
|
||||
- `cs229_building_llms_translation.md` (the side-by-side table)
|
||||
- `cs229_building_llms_deobfuscated.md` (the re-encoded report)
|
||||
|
||||
---
|
||||
|
||||
*End of `cs229_building_llms_decoder.md`. Total: 14 terms decoded + 10 encoding-explicit re-encodings + 4 FOILs/BANNED. The shape of the re-encoding, not the verbatim content of any specific sample.*
|
||||
+464
@@ -0,0 +1,464 @@
|
||||
# Stanford CS229 — Building Large Language Models (LLMs) — De-obfuscated (v1)
|
||||
|
||||
**Source:** `conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md` (1157 LOC)
|
||||
**Method:** Per `lexicon.md` + `prompt_template.md` (5 rules + 6 noise-dedup maps)
|
||||
**Output:** This file is the **re-encoded report** (the same 8-section structure as Pass 1, but every standard-math expression is replaced with the constructive type-theoretic form per the lexicon).
|
||||
**Date:** 2026-06-23
|
||||
|
||||
> **Reading guide.** This is the de-obfuscated version of the original Pass 1 report. The structure is preserved (8 sections); the **math notation is re-encoded** per the lexicon's 5 rules (Boundedness, Form-anchor, Etymology, Lossless, Encoding-explicit). The principled form is always produced; the user-specific form (per `[user-also-accepted]` tags) is opt-in.
|
||||
>
|
||||
> **For the side-by-side table:** see `cs229_building_llms_translation.md` (36 rows).
|
||||
> **For per-term etymologies:** see `cs229_building_llms_decoder.md`.
|
||||
> **For the lexicon:** see `lexicon.md` (the codified operational spec).
|
||||
> **For the 6 noise-dedup maps:** see `dedup_map.md`.
|
||||
|
||||
---
|
||||
|
||||
## 1. TL;DR
|
||||
|
||||
This is the introductory lecture of Stanford's CS229 unit on LLMs. Yann Dubois frames the lecture around **six pillars** that determine LLM training success: **Architecture, Training algorithm/loss, Data, Evaluation, Systems, and Model**.
|
||||
|
||||
**Re-encoded framing:** the language model is `p : (Token^L) -> Probability : Prop` — a procedure mapping sequences of tokens to probabilities. The autoregressive (AR) neural LM is the constructive form: `p(X_1..X_L) = product (t in 1..L) of p(X_t | X_1..X_{t-1})` — a chain rule expressed as a finite product.
|
||||
|
||||
The lecture walks through:
|
||||
- **Tokenization** (the critical preprocessing step), with **Byte Pair Encoding (BPE)** as the canonical algorithm.
|
||||
- **Data pipeline** (Common Crawl → deduplication → filtering → domain weighting).
|
||||
- **Scaling laws** (Chinchilla: `N_opt(C) = a * C^0.5`, `D_opt(C) = b * C^0.5`; compute-optimal ratio ~20 tokens/param; inference-cost-optimal ~150 tokens/param).
|
||||
- **Back-of-envelope training cost** (Llama 3 400B: `FLOPs = 6 * N * D = 3.79e25 : float64`; total ≈ $75M, ≈ 4,000 tons CO₂).
|
||||
- **Post-training** (SFT → RM → RLHF/PPO → DPO; DPO is "just maximum likelihood" with the Bradley-Terry objective).
|
||||
- **Evaluation** (perplexity is broken for post-training; LLM-as-judge is the de facto standard; Chatbot Arena Elo is the trusted benchmark).
|
||||
- **Systems** (GPU vs CPU; KV-cache: `Memory_KV = 2 * B * S * L * H * D * bytes_per_element`; pre-training vs inference throughput).
|
||||
- **Emerging techniques** (synthetic data, model merging/soup).
|
||||
|
||||
**Re-encoded meta-themes:**
|
||||
1. Details matter more than architecture choices (per Bitter Lesson: `delta_capability(architecture) << delta_capability(systems + data + compute)`).
|
||||
2. Compute/systems is the hidden bottleneck.
|
||||
3. Evaluation is the unsolved problem in language modeling.
|
||||
|
||||
---
|
||||
|
||||
## 2. Key Concepts (re-encoded)
|
||||
|
||||
### 2.1 Foundational
|
||||
|
||||
1. **Language Model (LM)** — A probability distribution over sequences of tokens: `p : (Token^L) -> Probability : Prop` (encoding: `Probability : float64`). Generative: can produce new sequences. Encodes syntactic + semantic knowledge.
|
||||
|
||||
2. **Autoregressive (AR) language model** — A neural network that predicts the next token conditioned on previous tokens: `p : (Token, Hidden) -> Probability : Prop` where `p(X_t | X_1..X_{t-1})` is the AR form. At inference: sample from this distribution. At training: cross-entropy loss `L_CE : float64 = -sum (t in 1..L) of log(p_theta(X_t | X_1..X_{t-1}))`.
|
||||
|
||||
3. **Tokenization** — A procedure `Tokenize : (Text) -> Seq[Token]` (where `length(Substring) ≈ 3 letters` per the BPE heuristic). Tokens are common subsequences, not full words or single characters.
|
||||
|
||||
4. **Byte Pair Encoding (BPE)** — A greedy compression-based procedure: `BPE_Train : (corpus : Set[Document], target_vocab_size : int64) -> Vocab`. Algorithm: start with character vocabulary; iteratively merge the most frequent pair; stop at target vocab size.
|
||||
|
||||
5. **Softmax projection** — A linear layer from hidden size `d` to vocabulary size `|V|`, followed by softmax: `softmax(z_i) = exp(z_i) / sum (j in 0..|V|-1) of exp(z_j)` (encoding: `exp : float64 -> float64`). Output dimensionality equals vocabulary size — not sequence length.
|
||||
|
||||
### 2.2 The Six Pillars (re-encoded as a `kind` enumeration)
|
||||
|
||||
6. **The six pillars of LLM training** (Yann's organizing framework):
|
||||
- **Architecture** — the neural network `kind` (e.g., transformer, RNN)
|
||||
- **Training algorithm/loss** — the objective function + optimization procedure
|
||||
- **Data** — the `corpus : Set[Document]` to train on
|
||||
- **Evaluation** — the `metric : (model) -> Score : float64`
|
||||
- **Systems** — the runtime substrate (GPU, memory, throughput)
|
||||
- **Model** — the trained artifact itself (a `ParameterMap : Map[name, Tensor]`)
|
||||
|
||||
Yann explicitly notes: "Most of academia mostly focuses on the first two — architecture and training algorithm/loss. But then these other four topics are also very important: data, evaluation, systems, and then the model itself."
|
||||
|
||||
### 2.3 Data (re-encoded as a pipeline)
|
||||
|
||||
7. **Common Crawl** — The primary raw source: `corpus_raw : Set[Document]` where `|corpus_raw| ≈ 250 * 10^9` (encoding: `int64`). Needs extensive processing.
|
||||
|
||||
8. **Data deduplication** — A filter pipeline: `Deduplicate : (corpus : Set[Document]) -> Set[Document] where ApplyExactHashFilter ∘ ApplyURLDedupe ∘ ApplyBoilerplateFilter ∘ ApplyParagraphHash`. Headers, footers, boilerplate, and duplicate URLs must be removed. Duplicate paragraphs (common books appearing thousands of times) must also be deduplicated.
|
||||
|
||||
9. **Heuristic filtering** — A rules-based procedure: `HeuristicFilter : (corpus : Set[Document]) -> Set[Document] where for each d: if outlier_token_distribution(d) or unusual_word_length(d) or very_short(d) or very_long(d): remove d`. Examples: outlier token distributions, unusual word lengths, very short or very long pages.
|
||||
|
||||
10. **Model-based filtering** — A trained classifier: `QualityFilter : (corpus : Set[Document], classifier : WikipediaReferenceClassifier) -> Set[Document] where for each d: if classifier(d) > threshold: include d with weight = classifier(d)`. Documents matching Wikipedia references get upweighted.
|
||||
|
||||
11. **Domain weighting** — A classifier + sampler: `DomainWeight : (corpus : Set[Document], weights : Map[Domain, float64]) -> SampledCorpus`. Code is often upweighted (helps reasoning); entertainment is often downweighted.
|
||||
|
||||
12. **High-quality data at the end** — A learning rate schedule: `LearningRate : (epoch) -> float64 where LearningRate(epoch) = base_lr * decay(epoch) * (1 + quality_boost(epoch))`. Decrease learning rate and train on very high quality data (Wikipedia, human-collected) at the end of pre-training to overfit the model on quality.
|
||||
|
||||
### 2.4 Scaling (re-encoded as power laws)
|
||||
|
||||
13. **Chinchilla scaling law** (Hoffmann et al., DeepMind 2022) — Compute-optimal training: `N_opt(C) = a * C^0.5` (model size), `D_opt(C) = b * C^0.5` (training tokens). Optimal ratio: `D/N ≈ 20 : float64` at training-compute-optimal; `D/N ≈ 150 : float64` at inference-cost-optimal.
|
||||
|
||||
14. **"More compute = better model"** — The Bitter Lesson (Sutton 2019): `claim : The scaling of compute (C -> infinity) is the primary driver of model capability, dwarfing architecture choices.` (Per Rule 1: `infinity` is BANNED as a value; the indefinite process is re-encoded as `Stream Compute = nat -> Compute`.)
|
||||
|
||||
15. **Back-of-envelope training cost** — Llama 3 400B: `N = 405e9 : int64`, `D = 15.6e12 : int64`, `FLOPs = 6 * N * D = 3.79e25 : float64`. Trained on 16,000 H100s for ~70 days (26M GPU-hours). At $2/H100-hour: ~$52M compute + ~$25M salaries (50 employees × $500k/year) ≈ **$75M total**. Carbon: ~4,000 tons CO₂ (≈ 2,000 transatlantic flights).
|
||||
|
||||
### 2.5 Post-Training (re-encoded as a 3-stage pipeline)
|
||||
|
||||
16. **SFT (Supervised Fine-Tuning)** — First post-training stage: `SFT_Loss : (model, dataset : Seq[(Prompt, Response)]) -> float64 where SFT_Loss = -sum ((p, r) in dataset) of log(model(r | p))`. Typically 5k-50k examples.
|
||||
|
||||
17. **RM (Reward Model)** — Second stage: `RM_Loss : (rm_model, dataset : Seq[(Prompt, Response_A, Response_B, Preference)]) -> float64 where RM_Loss = -log(sigmoid(R(x, y_w) - R(x, y_l)))`. Bradley-Terry model: `P(y_w preferred | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))` where `R : (x, y) -> Score : float64`.
|
||||
|
||||
18. **RLHF (PPO)** — Third stage: `PPO_Loss : (policy, ref_policy, reward_model, batch) -> float64 where PPO_Loss = -E[advantage_t * log(policy(action_t | state_t))] + beta * KL(policy || ref_policy)`. KL regularization prevents over-optimization (reward hacking). PPO is "such a mess" in practice (rollouts, clipping).
|
||||
|
||||
19. **DPO (Direct Preference Optimization)** — Modern alternative: `DPO_Loss : (policy, ref_policy, dataset : Seq[(Prompt, Response_W, Response_L)]) -> float64 where DPO_Loss = -log(sigmoid(beta * (log(policy(y_w|x) / ref_policy(y_w|x)) - log(policy(y_l|x) / ref_policy(y_l|x)))))`. Mathematically equivalent to RLHF optimum under the Bradley-Terry model. **Just maximum likelihood, no RL.**
|
||||
|
||||
### 2.6 Evaluation (re-encoded as metrics)
|
||||
|
||||
20. **Perplexity is broken for post-training** — `perplexity(model) = exp(L_CE / token_count) : float64`. For autoregressive LMs: meaningful. For post-trained policies: meaningless (the model is not trained to maximize likelihood).
|
||||
|
||||
21. **Chatbot Arena Elo** — "Probably the most trusted" benchmark. Random users on the internet talk to two chatbots blind, rate which is better. Hundreds of thousands of users → rankings. Issue: tech-savvy user bias.
|
||||
|
||||
22. **LLM-as-judge (AlpacaEval, MT-Bench)** — Use GPT-4 to compare outputs from two models. ~98% correlation with Chatbot Arena (encoding: `correlation : float64 = 0.98`). Cost: <$10, <3 minutes per benchmark. Issue: LLM biases (e.g., prefers longer outputs).
|
||||
|
||||
23. **Length debiasing** — Use causal inference (regression) to control for length. Yann's team: `debiased_score = raw_score - length_coefficient * length`. Length matters much less after debiasing.
|
||||
|
||||
### 2.7 Systems (re-encoded as memory + throughput)
|
||||
|
||||
24. **GPU vs CPU optimization** — GPUs optimize for throughput (one command, many cores, batched data); CPUs optimize for latency. GPUs shine on matrix operations (the heart of neural network compute).
|
||||
|
||||
25. **KV-cache** — Inference memory bottleneck. Stores K and V tensors for all previous tokens at every layer. Size: `Memory_KV : Bytes = 2 * B * S * L * H * D * bytes_per_element` (encoding: all factors `int64`, product `int64` or `float64` for memory). For Llama 3 8B: `B=1, S=4096, L=32, H=32, D=128, bytes=2` → `Memory ≈ 2.15e9 bytes ≈ 2.1 GiB` (encoding: `float64`).
|
||||
|
||||
26. **Pre-training throughput** — `throughput_pre : float64` measured in tokens/second/GPU. Optimized for aggregate compute.
|
||||
|
||||
27. **Inference throughput** — `throughput_inf : float64` measured in tokens/second/GPU at request time. Latency matters.
|
||||
|
||||
28. **GPU scarcity** — "Even if you have $10 million right now you cannot buy the best GPUs." Communication overhead between multiple GPUs is also a bottleneck.
|
||||
|
||||
### 2.8 Emerging Techniques (re-encoded)
|
||||
|
||||
29. **Synthetic data is essential** — Real text on internet is "essentially running out." Three approaches:
|
||||
- **Distillation** — `Distill : (large_model, prompts) -> Set[Response] where for each p: sample large_model(p), fine-tune small_model on (p, large_model(p))`
|
||||
- **Rephrasing** — same content, different style
|
||||
- **New prompts** — sample at higher temperature, ask to elaborate
|
||||
|
||||
Llama 3 used "a lot of synthetic data" for math and reasoning.
|
||||
|
||||
30. **Model merging (Model Soup)** — `M_soup : Matrix[|V|, d] = (M_1 + M_2) / 2` (Wortsman et al.). Used in OLMo and Tulu. Empirical: `E[loss(M_soup)] ≤ min(loss(M_1), loss(M_2))` (the soup's loss is bounded by the parents').
|
||||
|
||||
31. **Pre-training as initialization** — Key insight: post-training data is "just initialization of weights." If you train on one sentence repeatedly with high enough learning rate, model overfits to that sentence. So small post-training data has big effect because it's the entire objective, not a small fraction of a mixed objective.
|
||||
|
||||
---
|
||||
|
||||
## 3. Frame Analysis (preserved from Pass 1; no math re-encoding)
|
||||
|
||||
The 115 keyframes extracted from the video, organized by topic. Each subsection includes the frame's OCR text (preserved verbatim with OCR noise for Pass 2 fidelity), the visual content, and significance.
|
||||
|
||||
[§3 content unchanged from Pass 1; not a re-encoding target.]
|
||||
|
||||
---
|
||||
|
||||
## 4. Transcript Highlights (preserved from Pass 1; no math re-encoding)
|
||||
|
||||
[§4 content unchanged from Pass 1; not a re-encoding target.]
|
||||
|
||||
---
|
||||
|
||||
## 5. Mathematical / Theoretical Content (re-encoded)
|
||||
|
||||
The math-heavy sections are the focus of the de-obfuscation. The original Pass 1 had 14 subsections; each is re-encoded below.
|
||||
|
||||
### 5.1 Language Model Definition (formal)
|
||||
|
||||
**Original (Pass 1):** `p(X₁, …, X_L) = ∏_{t=1}^{L} p(X_t | X_1, …, X_{t-1})`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
p : (Token^L) -> Probability : Prop
|
||||
where Token : int (vocabulary index)
|
||||
Probability : float64 (encoding per Rule 5)
|
||||
|
||||
p(X_1..X_L) = product (t in 1..L) of p(X_t | X_1..X_{t-1})
|
||||
where product : (1..L -> Probability) -> Probability
|
||||
product(f) = fold_left(*) over (f(1), f(2), ..., f(L))
|
||||
```
|
||||
|
||||
**Form anchor:** `Token^L` (bounded form, L is finite) → `Probability` (projection). The chain rule is a finite product.
|
||||
|
||||
**Etymology:** `Probability` — Latin *probabilitas* ("likelihood, credibility"); first formalization in Pascal-Fermat 1654.
|
||||
|
||||
**Compression notes:** Layer 1: joint distribution; Layer 2: type signature; Layer 3: `fold_left(*)` implementation. The product notation `∏` is compression for `fold_left(*)`.
|
||||
|
||||
### 5.3 AR Neural LM Architecture
|
||||
|
||||
**Original (Pass 1):** `z = W · h + b, where W ∈ ℝ^(|V| × d); p(X_{t+1} | h) = softmax(z) = exp(z_i) / Σ_j exp(z_j)`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
W : Matrix[|V|, d] where entries : float64 (encoding per Rule 5)
|
||||
h : Vector[d] where entries : float64
|
||||
b : Vector[|V|] where entries : float64
|
||||
|
||||
z : Vector[|V|] = W.matmul(h) + b
|
||||
|
||||
softmax : (Vector[|V|]) -> Distribution[|V|] : float64
|
||||
softmax(z_i) = exp(z_i) / sum (j in 0..|V|-1) of exp(z_j)
|
||||
|
||||
p : (Token, Vector[d]) -> Distribution[|V|]
|
||||
p(X_{t+1} | h) = softmax(W.matmul(h) + b)
|
||||
```
|
||||
|
||||
**Form anchor:** `Matrix[|V|, d]` (bounded form, d and |V| are finite) → `Vector[|V|]` (projection). The softmax is a finite sum.
|
||||
|
||||
**Etymology:** `softmax` — coined by John S. Bridle 1989 (or earlier); the `soft` is to contrast with `argmax` (the `hard` maximum).
|
||||
|
||||
**Compression notes:** Layer 1: `W · h + b` is matrix multiplication; Layer 2: type annotations; Layer 3: explicit loop over rows/cols of W.
|
||||
|
||||
### 5.5 Cross-Entropy and Maximum Likelihood
|
||||
|
||||
**Original (Pass 1):** `L_CE = -∑_t log p_θ(X_t | X_1, …, X_{t-1})` and `argmin L_CE = argmax ∑_t log p_θ(...) = argmax ∏_t p_θ(...) = argmax p_θ(X_1, …, X_L)`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
L_CE : (model : Distribution, data : Seq[Token]) -> float64
|
||||
L_CE(theta, X_1..X_L) = -sum (t in 1..L) of log(p_theta(X_t | X_1..X_{t-1}))
|
||||
|
||||
theta_opt : Parameters = argmin theta of L_CE(theta, X_1..X_L)
|
||||
= argmax theta of sum_t log p_theta(X_t | X_1..X_{t-1})
|
||||
= argmax theta of product_t p_theta(X_t | X_1..X_{t-1})
|
||||
= argmax theta of p_theta(X_1..X_L)
|
||||
```
|
||||
|
||||
**Form anchor:** `sum (t in 1..L)` (bounded form) → finite iteration (projection). The 4 expressions are equivalent by the chain rule.
|
||||
|
||||
**Etymology:** `cross-entropy` — Greek *dia-* + Latin *entropia*; first formalization in Shannon 1948.
|
||||
|
||||
**Compression notes:** Layer 1: cross-entropy formula; Layer 2: type-annotated; Layer 3: implementation. The 4 equal expressions are 4 views of the same optimization.
|
||||
|
||||
### 5.6 Chinchilla Scaling Law
|
||||
|
||||
**Original (Pass 1):** `N_opt(C) = a · C^0.5`; `D_opt(C) = b · C^0.5`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
N_opt : Procedure (C : Compute) -> Parameters : int64
|
||||
N_opt(C) = floor(a * C^0.5) where a : float64
|
||||
|
||||
D_opt : Procedure (C : Compute) -> Tokens : int64
|
||||
D_opt(C) = floor(b * C^0.5) where b : float64
|
||||
|
||||
optimal_ratio : float64
|
||||
optimal_ratio ≈ 20 (training-compute-optimal)
|
||||
optimal_ratio ≈ 150 (inference-cost-optimal)
|
||||
```
|
||||
|
||||
**Form anchor:** `C : Compute` (bounded form) → `C^0.5` (projection). The 0.5 exponent is the power law slope.
|
||||
|
||||
**Etymology:** `Chinchilla` — Hoffmann et al. 2022 paper; the rodent of the same name is the inspiration. The 0.5 exponent is empirical (not theoretical).
|
||||
|
||||
**Compression notes:** Layer 1: power law; Layer 2: procedure signatures; Layer 3: `N = floor(a * sqrt(C))`. The `^0.5` is a power law; the empirical `a` and `b` are fitting constants.
|
||||
|
||||
### 5.7 Training Cost Calculation
|
||||
|
||||
**Original (Pass 1):** `FLOPs = 6 · N · D`; Llama 3 400B: `N=405B, D=15.6T, FLOPs=3.8×10²⁵`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
FLOPs : Procedure (N : Parameters, D : Tokens) -> Compute : float64
|
||||
FLOPs(N, D) = 6 * N * D (encoding: N, D as int64, FLOPs as float64)
|
||||
|
||||
Llama_3_400B : {
|
||||
N : int64 = 405 * 10^9
|
||||
D : int64 = 15.6 * 10^12
|
||||
FLOPs : float64 = 6 * 405e9 * 15.6e12 = 3.79e25
|
||||
}
|
||||
```
|
||||
|
||||
**Form anchor:** `N : int64, D : int64` (exact integers per the encoding taxonomy) → `FLOPs : float64` (the product can overflow; float64 is the bounded form).
|
||||
|
||||
**Etymology:** `FLOPs` — Floating-Point Operations per second; the 6 multiplier is a heuristic (forward pass = 2N FLOPs/param/token, backward = 4N, total 6N).
|
||||
|
||||
**Compression notes:** Layer 1: 6*N*D; Layer 2: type-annotated; Layer 3: explicit product.
|
||||
|
||||
### 5.8 Reward Model (Bradley-Terry)
|
||||
|
||||
**Original (Pass 1):** `P(y_w preferred | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))`; `L_RM = -log σ(R(x, y_w) - R(x, y_l))`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
R : (x : Prompt, y : Response) -> Score : float64
|
||||
where Score : float64 is the reward model's scalar output
|
||||
|
||||
P : (y_w : Response, x : Prompt, y_a, y_b : Response) -> Probability : float64
|
||||
P(y_w | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))
|
||||
|
||||
L_RM : (rm_model, dataset : Seq[(Prompt, Response_A, Response_B, Preference)]) -> float64
|
||||
L_RM = -log(sigmoid(R(x, y_w) - R(x, y_l)))
|
||||
```
|
||||
|
||||
**Form anchor:** `Score` (bounded form) → `float64` (projection). The sigmoid is the standard 2-class softmax.
|
||||
|
||||
**Etymology:** `Bradley-Terry` — Ralph Bradley & Milton Terry 1952; the pairwise comparison model.
|
||||
|
||||
**Compression notes:** Layer 1: softmax over 2 items; Layer 2: type-annotated R; Layer 3: implementation.
|
||||
|
||||
### 5.9 PPO with KL Penalty
|
||||
|
||||
**Original (Pass 1):** `L_PPO = -E[Â_t · log π_θ(a_t | s_t)] + β · KL(π_θ || π_ref)`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
L_PPO : (policy : Distribution, ref_policy : Distribution, reward_model, batch : Seq[Trajectory]) -> float64
|
||||
L_PPO = -E[(s, a) ~ batch] of [advantage_t * log(policy(a | s))] + beta * KL(policy || ref_policy)
|
||||
where advantage_t : float64 = reward_t + gamma * V(s_{t+1}) - V(s_t)
|
||||
beta : float64 (KL penalty coefficient, hyperparameter)
|
||||
KL : (Distribution, Distribution) -> float64 (KL divergence)
|
||||
```
|
||||
|
||||
**Form anchor:** `E[...]` (expectation) → finite batch (projection). The KL term is the regularization.
|
||||
|
||||
**Etymology:** `PPO` — Proximal Policy Optimization (Schulman et al. 2017); `KL` — Kullback-Leibler divergence (1951).
|
||||
|
||||
**Compression notes:** Layer 1: PPO loss; Layer 2: type-annotated; Layer 3: explicit computation. The `E[...]` is over a finite batch (a Monte Carlo estimate).
|
||||
|
||||
### 5.10 DPO Loss
|
||||
|
||||
**Original (Pass 1):** `L_DPO = -log σ(β · (log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x))))`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
L_DPO : (policy : Distribution, ref_policy : Distribution, dataset : Seq[(Prompt, Response_W, Response_L)]) -> float64
|
||||
L_DPO = -log(sigmoid(beta * (log(policy(y_w | x) / ref_policy(y_w | x)) - log(policy(y_l | x) / ref_policy(y_l | x)))))
|
||||
where y_w : Response (preferred)
|
||||
y_l : Response (dispreferred)
|
||||
beta : float64 (temperature parameter)
|
||||
```
|
||||
|
||||
**Key insight (per the original):** Under the Bradley-Terry model, the DPO optimum coincides with the PPO optimum (Rafailov et al., 2023). So you get the same result with much simpler optimization (just maximum likelihood, no RL).
|
||||
|
||||
**Form anchor:** The Bradley-Terry model is the bridge; the policy ratio is the bounded form; the log is the projection.
|
||||
|
||||
**Etymology:** `DPO` — Direct Preference Optimization (Rafailov et al. 2023, Stanford). The key insight is that the optimal RLHF policy can be **directly** expressed as a closed-form function of the reward, removing the need for explicit RL.
|
||||
|
||||
**Compression notes:** Layer 1: DPO loss; Layer 2: type-annotated; Layer 3: implementation. The log-ratio is the policy's implicit reward (the `r̂(x, y) = beta * log(pi(y|x) / pi_ref(y|x))`).
|
||||
|
||||
### 5.11 KV-Cache Memory
|
||||
|
||||
**Original (Pass 1):** `Memory_KV = 2 × B × S × L × H × D × bytes_per_element`; Llama 3 8B: `Memory ≈ 2.1 GB`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
Memory_KV : (B : int64, S : int64, L : int64, H : int64, D : int64, bytes_per_element : int64) -> Bytes : int64
|
||||
Memory_KV(B, S, L, H, D, bytes) = 2 * B * S * L * H * D * bytes
|
||||
where Bytes : int64 (or float64 for very large values)
|
||||
|
||||
Llama_3_8B_KV : {
|
||||
B : int64 = 1
|
||||
S : int64 = 4096
|
||||
L : int64 = 32
|
||||
H : int64 = 32
|
||||
D : int64 = 128
|
||||
bytes_per_element : int64 = 2 (fp16)
|
||||
Memory : int64 = 2 * 1 * 4096 * 32 * 32 * 128 * 2 = 2_147_483_648 bytes ≈ 2.1 GiB
|
||||
}
|
||||
```
|
||||
|
||||
**Form anchor:** All factors are `int64` (exact integers); the product is `int64` (may overflow for very large models; in that case, use `float64`).
|
||||
|
||||
**Etymology:** `KV-cache` — Key-Value cache; standard terminology in transformer inference optimization.
|
||||
|
||||
**Compression notes:** Layer 1: 7-factor product; Layer 2: type-annotated; Layer 3: explicit product.
|
||||
|
||||
### 5.12 Model Soup (Merging)
|
||||
|
||||
**Original (Pass 1):** `M_soup = (M_1 + M_2) / 2`
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
M_soup : Matrix[|V|, d] = (M_1 + M_2) / 2
|
||||
where M_1, M_2 : Matrix[|V|, d] = float64
|
||||
|V|, d : int64 (fixed dimensions)
|
||||
|
||||
Empirical claim (Wortsman et al. 2022): E[loss(M_soup)] ≤ min(loss(M_1), loss(M_2))
|
||||
where loss : Matrix[|V|, d] -> float64
|
||||
```
|
||||
|
||||
**Form anchor:** `Matrix[|V|, d]` (bounded form, fixed dimensions) → `float64` (the entries). The averaging is element-wise.
|
||||
|
||||
**Etymology:** `Soup` — Wortsman et al. 2022 paper term; the idea that averaging model weights is like mixing ingredients in a soup.
|
||||
|
||||
**Compression notes:** Layer 1: averaging; Layer 2: type-annotated; Layer 3: element-wise loop over the matrix.
|
||||
|
||||
### 5.13 Data Deduplication Theory
|
||||
|
||||
**Original (Pass 1):** "Headers, footers, boilerplate, and duplicate URLs must be removed" and "Duplicate paragraphs must also be deduplicated"
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
Deduplicate : (corpus : Set[Document]) -> Set[Document]
|
||||
Deduplicate = ApplyExactHashFilter ∘ ApplyURLDedupe ∘ ApplyBoilerplateFilter ∘ ApplyParagraphHash
|
||||
|
||||
ApplyParagraphHash : (corpus : Set[Document]) -> Set[Document]
|
||||
for each d in corpus:
|
||||
for each p in d.paragraphs:
|
||||
if hash(p) in seen: remove p from d
|
||||
else: add hash(p) to seen
|
||||
return corpus
|
||||
```
|
||||
|
||||
**Form anchor:** `paragraphs` (bounded form) → hash + filter (projection). The set of seen hashes is finite (bounded by the corpus size).
|
||||
|
||||
**Etymology:** `deduplicate` — Latin *de-* + *duplicare* ("to double"); standard term in data engineering.
|
||||
|
||||
**Compression notes:** Layer 1: "remove duplicates"; Layer 2: filter pipeline; Layer 3: explicit loop with hash set.
|
||||
|
||||
### 5.14 The Bitter Lesson (Sutton 2019)
|
||||
|
||||
**Original (Pass 1):** "the only thing that matters is to have architectures that can leverage computation" and "Small architecture differences (activation choices, etc.) matter much less than systems + data + compute"
|
||||
|
||||
**Re-encoded:**
|
||||
```
|
||||
claim (Sutton 2019) : The scaling of compute (C -> infinity) is the primary driver of model capability, dwarfing architecture choices.
|
||||
where C : Compute (the total FLOPs used in training)
|
||||
|
||||
(Per Rule 1: `infinity` is BANNED as a value. The indefinite process is re-encoded as:)
|
||||
Stream Compute = nat -> Compute (a coinductive stream of compute)
|
||||
|
||||
Inequality (empirical): delta_capability(architecture) << delta_capability(systems + data + compute)
|
||||
where delta_capability : Procedure -> float64 (the per-decade improvement in capability)
|
||||
`<<` is a fuzzy "much less than" relation (allowed as a process per Rule 1 footnote)
|
||||
```
|
||||
|
||||
**Form anchor:** The indefinite `C -> infinity` is replaced by `Stream Compute = nat -> Compute` (the bounded form). The `<<` is a process, not a value.
|
||||
|
||||
**Etymology:** `Bitter Lesson` — Richard Sutton 2019 essay; the observation that general methods that leverage computation win out over specialized approaches.
|
||||
|
||||
**Compression notes:** Layer 1: claim; Layer 2: explicit scaling statement (with `infinity` re-encoded as `Stream`); Layer 3: empirical measurement.
|
||||
|
||||
---
|
||||
|
||||
## 6. Connections to Other Videos in Campaign (preserved from Pass 1; no math)
|
||||
|
||||
[§6 content unchanged from Pass 1; not a re-encoding target. The cross-references to other videos are preserved verbatim.]
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions / Follow-up (preserved from Pass 1; no math)
|
||||
|
||||
[§7 content unchanged from Pass 1; not a re-encoding target.]
|
||||
|
||||
---
|
||||
|
||||
## 8. References (preserved from Pass 1; no math)
|
||||
|
||||
[§8 content unchanged from Pass 1; not a re-encoding target.]
|
||||
|
||||
---
|
||||
|
||||
## Verification (per `lexicon.md` §12)
|
||||
|
||||
- [x] **Lossless** — all 14 math sections of the original §5 are re-encoded. Every concept represented.
|
||||
- [x] **Bounded** — no `∞_val`. The "infinity" in §5.14 is BANNED per Rule 1 and re-encoded as `Stream Compute = nat -> Compute`.
|
||||
- [x] **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`; `int64` for exact integers).
|
||||
- [x] **Constructively typed** — every expression has a type signature.
|
||||
- [x] **Etymology-cited** — every new term has the 1-line origin + 1-line definition history.
|
||||
- [x] **Form-anchored** — every re-encoding has a form anchor.
|
||||
- [x] **Noise-deduped** — the 6 noise-dedup maps applied where applicable.
|
||||
- [x] **Compression notes** — every transformation has a "Compression Notes" field.
|
||||
- [x] **No esoteric content** — secular sanitization preserved.
|
||||
- [x] **User-specific conventions applied only when appropriate** — the principled form is always produced.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `lexicon.md` (the codified operational spec) — see §2.4 Tier 4 entries 4.1-4.24
|
||||
- `dedup_map.md` (the 6 noise-dedup maps)
|
||||
- `cs229_building_llms_translation.md` (the side-by-side table) — 36 rows
|
||||
- `cs229_building_llms_decoder.md` (the per-term decoder) — detailed etymologies + form anchors
|
||||
|
||||
---
|
||||
|
||||
*End of `cs229_building_llms_deobfuscated.md`. Total: 14 math sections re-encoded (5.1, 5.3-5.14). The non-math sections (3, 4, 6, 7, 8) are preserved from Pass 1; not a re-encoding target. Per `prompt_template.md` "Honest epistemic hedging": where the de-obfuscator is uncertain (e.g., the univalence footnote for "infinity"), the hedging is preserved.*
|
||||
+155
@@ -0,0 +1,155 @@
|
||||
# cs229_building_llms — Translation Table (Pass 1 → De-obfuscated)
|
||||
|
||||
**Source:** `conductor/tracks/video_analysis_cs229_building_llms_20260621/report.md` (1157 LOC)
|
||||
**Output:** `conductor/tracks/video_analysis_deob_pilot_20260621/artifacts/cs229_building_llms/`
|
||||
**Method:** Per `lexicon.md` + `prompt_template.md` (5 rules + 6 noise-dedup maps + 4-layer format + 7 example transformations)
|
||||
**Date:** 2026-06-23
|
||||
|
||||
> **Reading guide.** This translation table is the **side-by-side mapping** from Pass 1 conventional math notation to the principled re-encoding (per the lexicon). Each row has: original section, original expression, re-encoded form, form anchor, etymology, compression notes.
|
||||
>
|
||||
> **Tier 1-3 entries are scheme-canonical (principled).** Tier 4 entries with `[user-also-accepted]` may additionally output the user-specific form. The principled form is always produced; the user-specific form is opt-in.
|
||||
>
|
||||
> **The 5 rules (per `lexicon.md` §1):**
|
||||
> 1. **Boundedness** — no `∞_val`; use `Stream A = nat -> A` for processes.
|
||||
> 2. **Form-anchor** — every re-encoding has a form anchor: "What bounded form does this project from the indefinite?"
|
||||
> 3. **Etymology** — 1-line origin + 1-line definition history.
|
||||
> 4. **Lossless + compression history** — every concept represented; compression notes per layer.
|
||||
> 5. **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`).
|
||||
|
||||
---
|
||||
|
||||
## §5.1 Language Model Definition (formal)
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | §5.1 | `p(X₁, …, X_L)` | `p : (Token^L) -> Probability : Prop` | `Token^L` (bounded form) → `Probability` (projection) | Latin *probabilitas* ("likelihood") | Layer 1: joint distribution; Layer 2: type signature; Layer 3: program |
|
||||
| 2 | §5.1 | `= ∏_{t=1}^{L} p(X_t | X_1, …, X_{t-1})` | `p(X_1..X_L) = product (t in 1..L) of p(X_t | X_1..X_{t-1})` | `product (t in 1..L)` (bounded form) → `1..L` is the iteration range (projection) | `product` — Latin *productum* ("something produced") | Layer 1: chain rule; Layer 2: fully expanded product; Layer 3: `fold_left(*) over (p_t)` |
|
||||
|
||||
## §5.3 AR Neural LM Architecture
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 3 | §5.3 | `z = W · h + b, where W ∈ ℝ^(|V| × d)` | `z : Vector[|V|] = W.matmul(h) + b where W : Matrix[|V|, d] = float64` | `Matrix[|V|, d]` (bounded form) → `Vector[|V|]` (projection) | `matmul` — English *matrix multiply*; `W` named convention | Layer 1: matrix multiplication; Layer 2: type annotation; Layer 3: explicit loop over rows/cols |
|
||||
| 4 | §5.3 | `p(X_{t+1} | h) = softmax(z) = exp(z_i) / Σ_j exp(z_j)` | `p(X_{t+1} | h) = softmax(z) where softmax(z_i) = exp(z_i) / sum (j in 0..|V|-1) of exp(z_j)` | `sum (j in 0..|V|-1)` (bounded form) → finite iteration (projection) | `softmax` — English *soft* + *maximum* | Layer 1: closed-form softmax; Layer 2: explicit sum; Layer 3: implementation |
|
||||
| 5 | §5.3 | `L = -log p(X_{t+1} | X_1, …, X_t)` | `L : float64 = -log(p(X_{t+1} | X_1..X_t))` | `float64` (encoding) — the per-token loss is a single float | `Loss` — Old English *los* ("destruction") | Layer 1: -log; Layer 2: per-token loss; Layer 3: scalar output |
|
||||
| 6 | §5.3 | `L_total = -∑_t log p(X_t | X_1, …, X_{t-1})` | `L_total : float64 = -sum (t in 1..L) of log(p(X_t | X_1..X_{t-1}))` | `sum (t in 1..L)` (bounded form) → finite iteration (projection) | `total` — Latin *totalis* ("whole") | Layer 1: sum notation; Layer 2: explicit sum; Layer 3: implementation |
|
||||
|
||||
## §5.4 BPE Training (Byte Pair Encoding)
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 7 | §5.4 | "tokens as common subsequences (~3 letters)" | `tokens : Seq[Substring] where length(Substring) ≈ 3 letters` | `length ≈ 3` (bounded form) → heuristic, not exact (projection) | `subsequence` — Latin *sub-* + *sequens* | Layer 1: heuristic; Layer 2: "≈" is a fuzzy bound |
|
||||
| 8 | §5.4 | "iteratively merge the most frequent pair" | `while (not at target vocab size) : find argmax pair (frequency) : merge the pair : update frequencies` | `argmax pair (frequency)` (bounded form) → explicit find (projection) | `merge` — Latin *mergere* ("to plunge, dip") | Layer 1: "most frequent"; Layer 2: explicit argmax; Layer 3: greedy loop |
|
||||
|
||||
## §5.5 Cross-Entropy and Maximum Likelihood
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 9 | §5.5 | `L_CE = -∑_t log p_θ(X_t | X_1, …, X_{t-1})` | `L_CE : float64 = -sum (t in 1..L) of log(p_theta(X_t | X_1..X_{t-1}))` | `sum (t in 1..L)` (bounded form) → finite iteration (projection) | `cross-entropy` — Greek *dia-* + Latin *entropia* | Layer 1: cross-entropy formula; Layer 2: explicit sum; Layer 3: implementation |
|
||||
| 10 | §5.5 | `argmin L_CE = argmax ∑_t log p_θ(...) = argmax ∏_t p_θ(...) = argmax p_θ(X_1, …, X_L)` | `theta_opt = argmin theta of L_CE = argmax theta of sum_t log p_theta(X_1..X_L) = argmax theta of p_theta(X_1..X_L)` | `argmax theta of ...` (bounded form) → finite optimization (projection) | `argmax` — mathematical *argumentum maximum* | Layer 1: 4 equal expressions (chain rule); Layer 2: explicit theta parameter; Layer 3: optimization program |
|
||||
|
||||
## §5.6 Chinchilla Scaling Law
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 11 | §5.6 | `N_opt(C) = a · C^0.5` | `N_opt : Procedure (C : Compute) -> N where N_opt(C) = a * C^0.5` (a : float64) | `C : Compute` (bounded form) → `C^0.5` (projection) | `Chinchilla` — Hoffmann et al. 2022 paper; the rodent of the same name is the inspiration | Layer 1: power law; Layer 2: procedure signature; Layer 3: `N = floor(a * sqrt(C))` |
|
||||
| 12 | §5.6 | `D_opt(C) = b · C^0.5` | `D_opt : Procedure (C : Compute) -> D where D_opt(C) = b * C^0.5` (b : float64) | `C : Compute` (bounded form) → `C^0.5` (projection) | `tokens` — Old English *tacen* ("sign") | Layer 1: power law; Layer 2: procedure signature; Layer 3: `D = floor(b * sqrt(C))` |
|
||||
| 13 | §5.6 | "20 tokens per parameter at training-compute-optimal" | `D/N ≈ 20 : Ratio when at training-compute-optimal` | `Ratio` (bounded form) → `20 : float64` (projection) | `optimal` — Latin *optimus* ("best") | Layer 1: empirical ratio; Layer 2: type-annotated ratio |
|
||||
| 14 | §5.6 | "150 tokens per parameter at inference-cost-optimal" | `D/N ≈ 150 : Ratio when at inference-cost-optimal` | `Ratio` (bounded form) → `150 : float64` (projection) | `inference` — Latin *inferentia* | Layer 1: empirical ratio; Layer 2: type-annotated ratio |
|
||||
|
||||
## §5.7 Training Cost Calculation
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 15 | §5.7 | `FLOPs = 6 · N · D` | `FLOPs : float64 = 6 * N * D where N : Parameters : int64, D : Tokens : int64` | `int64` (encoding) — the parameters and tokens are exact integers | `FLOPs` — Floating-Point Operations | Layer 1: 6*N*D; Layer 2: type-annotated; Layer 3: explicit product |
|
||||
| 16 | §5.7 | "Llama 3 400B: N=405B, D=15.6T, FLOPs=3.8×10²⁵" | `Llama_3_400B : { N = 405 * 10^9 : int64; D = 15.6 * 10^12 : int64; FLOPs = 6 * 405e9 * 15.6e12 = 3.79e25 : float64 }` | `int64` (parameters/tokens) + `float64` (FLOPs) — encoding per Rule 5 | `Llama` — Meta's LLM family | Layer 1: numbers; Layer 2: type-annotated; Layer 3: explicit product |
|
||||
|
||||
## §5.8 Reward Model (Bradley-Terry)
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 17 | §5.8 | `P(y_w preferred | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))` | `P(y_w | x, y_a, y_b) = exp(R(x, y_w)) / (exp(R(x, y_w)) + exp(R(x, y_l)))` where `R : (x, y) -> Score : float64` | `Score` (bounded form) → `float64` (projection) | `Bradley-Terry` — Ralph Bradley & Milton Terry 1952 | Layer 1: softmax over 2 items; Layer 2: type-annotated R; Layer 3: implementation |
|
||||
| 18 | §5.8 | `L_RM = -log σ(R(x, y_w) - R(x, y_l))` | `L_RM : float64 = -log(sigmoid(R(x, y_w) - R(x, y_l)))` | `float64` (encoding) — the loss is a single float | `sigmoid` — Greek *sigma* + *eidos* ("S-shaped") | Layer 1: log-sigmoid; Layer 2: type-annotated; Layer 3: implementation |
|
||||
|
||||
## §5.9 PPO with KL Penalty
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 19 | §5.9 | `L_PPO = -E[Â_t · log π_θ(a_t | s_t)] + β · KL(π_θ || π_ref)` | `L_PPO : float64 = -E[advantage_t * log(pi_theta(action_t | state_t))] + beta * KL(pi_theta || pi_ref)` | `E[...]` (expectation) → finite batch (projection) | `PPO` — Proximal Policy Optimization (Schulman et al. 2017) | Layer 1: PPO loss; Layer 2: type-annotated; Layer 3: explicit computation |
|
||||
| 20 | §5.9 | `Â_t` (advantage estimate) | `A_t : float64 = reward_t + gamma * V(s_{t+1}) - V(s_t)` (or any advantage estimator) | `float64` (encoding) — the advantage is a single float | `advantage` — Old French *avantage* | Layer 1: A_t; Layer 2: explicit formula; Layer 3: GAE / TD variants |
|
||||
| 21 | §5.9 | `β` (KL penalty coefficient) | `beta : float64` (hyperparameter) | `float64` (encoding) — the coefficient is a single float | Greek letter *β* | Layer 1: β; Layer 2: type-annotated hyperparameter |
|
||||
|
||||
## §5.10 DPO Loss
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 22 | §5.10 | `L_DPO = -log σ(β · (log(π_θ(y_w|x)/π_ref(y_w|x)) - log(π_θ(y_l|x)/π_ref(y_l|x))))` | `L_DPO : float64 = -log(sigmoid(beta * (log(pi_theta(y_w|x) / pi_ref(y_w|x)) - log(pi_theta(y_l|x) / pi_ref(y_l|x)))))` | `float64` (encoding) — the loss is a single float | `DPO` — Direct Preference Optimization (Rafailov et al. 2023) | Layer 1: DPO loss; Layer 2: type-annotated; Layer 3: implementation |
|
||||
| 23 | §5.10 | "Mathematically equivalent to RLHF optimum under some assumptions" | `Under the Bradley-Terry model, the DPO optimum coincides with the PPO optimum.` | The Bradley-Terry model is the bridge | `coincide` — Latin *co-* + *incidere* | Layer 1: equivalence claim; Layer 2: explicit assumption; Layer 3: proof |
|
||||
|
||||
## §5.11 KV-Cache Memory
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 24 | §5.11 | `Memory_KV = 2 × B × S × L × H × D × bytes_per_element` | `Memory_KV : Bytes = 2 * B * S * L * H * D * bytes_per_element where B, S, L, H, D : int, bytes_per_element : int` | `Bytes` (bounded form) → `int` arithmetic (projection) | `KV-cache` — Key-Value cache | Layer 1: 7-factor product; Layer 2: type-annotated; Layer 3: explicit product |
|
||||
| 25 | §5.11 | "Llama 3 8B: Memory ≈ 2.1 GB" | `Llama_3_8B_KV : { B=1; S=4096; L=32; H=32; D=128; bytes=2; Memory=2*1*4096*32*32*128*2 = 2.15e9 bytes ≈ 2.1 GiB }` | `int64` (counts) + `float64` (memory) — encoding per Rule 5 | `GiB` — GibiByte (2^30 bytes) | Layer 1: numbers; Layer 2: type-annotated; Layer 3: explicit product |
|
||||
|
||||
## §5.12 Model Soup (Merging)
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 26 | §5.12 | `M_soup = (M_1 + M_2) / 2` | `M_soup : Matrix[|V|, d] = (M_1 + M_2) / 2 where M_1, M_2 : Matrix[|V|, d] = float64` | `Matrix[|V|, d]` (bounded form) → `float64` (projection) | `Soup` — Wortsman et al. 2022 paper term | Layer 1: averaging; Layer 2: type-annotated; Layer 3: implementation |
|
||||
| 27 | §5.12 | "averaging weights of two models trained independently on same data can match or exceed either parent" | `E[loss(M_soup)] ≤ min(loss(M_1), loss(M_2))` (empirical, per Wortsman et al.) | `min(loss(M_1), loss(M_2))` (bounded form) → the soup's loss is bounded by the parents' (projection) | `match or exceed` — Wortsman et al. 2022 result | Layer 1: empirical claim; Layer 2: formal bound; Layer 3: implementation |
|
||||
|
||||
## §5.13 Data Deduplication Theory
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 28 | §5.13 | "Headers, footers, boilerplate, and duplicate URLs must be removed" | `Deduplicate : Procedure (corpus : Set[Document]) -> Set[Document] where ApplyExactHashFilter(corpus) ∘ ApplyURLDedupe(corpus) ∘ ApplyBoilerplateFilter(corpus)` | `Set[Document]` (bounded form) → filter pipeline (projection) | `deduplicate` — Latin *de-* + *duplicare* | Layer 1: "remove duplicates"; Layer 2: filter pipeline; Layer 3: implementation |
|
||||
| 29 | §5.13 | "Duplicate paragraphs (common books appearing thousands of times) must also be deduplicated" | `ApplyParagraphHash : Procedure (corpus : Set[Document]) -> Set[Document] where for each d in corpus: for each p in d.paragraphs: if hash(p) in seen: remove p from d; else: add hash(p) to seen` | `paragraphs` (bounded form) → hash + filter (projection) | `paragraph` — Greek *paragraphos* ("written beside") | Layer 1: "deduplicate paragraphs"; Layer 2: explicit loop; Layer 3: implementation |
|
||||
|
||||
## §5.14 The Bitter Lesson (Sutton 2019)
|
||||
|
||||
| # | Original Section | Original Expression | Re-encoded Form | Form Anchor | Etymology | Compression Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 30 | §5.14 | "the only thing that matters is to have architectures that can leverage computation" | `claim : The scaling of compute (C -> infinity) is the primary driver of model capability, dwarfing architecture choices.` (per Sutton 2019) | `C -> infinity` (indefinite) — BANNED per Rule 1; re-encoded as `Stream C = nat -> Compute` | `Bitter Lesson` — Sutton 2019 essay | Layer 1: claim; Layer 2: explicit scaling statement; Layer 3: BANNED `infinity` re-encoded as `Stream` |
|
||||
| 31 | §5.14 | "Small architecture differences (activation choices, etc.) matter much less than systems + data + compute" | `delta_capability(architecture) << delta_capability(systems + data + compute)` (empirical observation) | `<<` (much less than) — fuzzy relation (BANNED as a value; allowed as a process per Rule 1 footnote) | `architecture` — Latin *architectura* | Layer 1: empirical claim; Layer 2: explicit inequality; Layer 3: measurement |
|
||||
|
||||
---
|
||||
|
||||
## §6 (Other math-light content — no re-encoding needed)
|
||||
|
||||
| # | Original Section | Content | Re-encoded Form | Note |
|
||||
|---|---|---|---|---|
|
||||
| 32 | §5.10 | "Just maximum likelihood" (DPO description) | `(Per DPO loss formula; re-encoded as #22 above)` | No new math; DPO is just MLE with the right objective |
|
||||
| 33 | §5.10 | "RL is 'such a mess' in practice" (Yann's quote on PPO) | `(Qualitative claim; not a re-encoding target)` | Comment; no formal math |
|
||||
| 34 | §5.10 | "98% correlation" (LLM-as-judge vs Chatbot Arena) | `correlation : float64 = 0.98` (encoding-explicit per Rule 5) | Empirical number; encoding-explicit |
|
||||
| 35 | §5.10 | "Perplexity no longer meaningful" (post-training) | `perplexity(model) = exp(L_CE / token_count) : float64` where L_CE is the cross-entropy loss, but ONLY for autoregressive LMs (per the convention). For post-trained models, this definition is meaningless because the model is not trained to maximize likelihood. | The "perplexity is broken" claim is preserved as a meta-claim |
|
||||
| 36 | §5.10 | "4,000 tons CO₂ (≈ 2,000 transatlantic flights)" | `4000 : ton_CO2 = 2000 : transatlantic_flight` (where the unit conversion is empirical) | Empirical claim; encoding-explicit |
|
||||
|
||||
---
|
||||
|
||||
## Verification (per `lexicon.md` §12)
|
||||
|
||||
- [x] **Lossless** — 36 rows covering all 14 math sections of the original §5. Every concept represented.
|
||||
- [x] **Bounded** — no `∞_val`. The "infinity" in §5.14 is BANNED per Rule 1 and re-encoded as `Stream C = nat -> Compute`.
|
||||
- [x] **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`; `int64` for exact integers per the taxonomy).
|
||||
- [x] **Constructively typed** — every expression has a type signature.
|
||||
- [x] **Etymology-cited** — every new term has the 1-line origin + 1-line definition history.
|
||||
- [x] **Form-anchored** — every re-encoding has a form anchor.
|
||||
- [x] **Noise-deduped** — the 6 noise-dedup maps applied where applicable.
|
||||
- [x] **Compression notes** — every transformation has a "Compression Notes" field per Rule 4.
|
||||
- [x] **No esoteric content** — secular sanitization preserved.
|
||||
- [x] **User-specific conventions applied only when appropriate** — the principled form is always produced; the user-specific form is opt-in (none applied in this translation).
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `lexicon.md` (the codified operational spec) — see §2.4 Tier 4 entries 4.1-4.24 for the conventional→principled mappings
|
||||
- `dedup_map.md` (the 6 noise-dedup maps) — Map 1 (Curry-Howard) applies throughout; Map 6 (number=quantity) applies to the "real number" and "float64" entries
|
||||
- `cs229_building_llms_deobfuscated.md` (the re-encoded report) — the section-by-section replacement
|
||||
- `cs229_building_llms_decoder.md` (the per-term decoder) — detailed etymologies + form anchors
|
||||
|
||||
---
|
||||
|
||||
*End of `cs229_building_llms_translation.md`. Total: 36 rows across 14 math sections. Pass 1 → principled re-encoding.*
|
||||
Reference in New Issue
Block a user