conductor(deob_apply): platonic_intelligence_kumar deobfuscated (12 math sections in §5 re-encoded, Stream replaces ∞_val, full compression notes)
This commit is contained in:
+455
@@ -0,0 +1,455 @@
|
||||
# Towards a Platonic Intelligence with Unified Factored Representations — De-obfuscated
|
||||
|
||||
**Source:** `conductor/tracks/video_analysis_platonic_intelligence_kumar_20260621/report.md` (1563 LOC)
|
||||
**Output:** `conductor/tracks/video_analysis_deob_apply_20260621/artifacts/platonic_intelligence_kumar/platonic_intelligence_kumar_deobfuscated.md`
|
||||
**Method:** Per `lexicon.md` (5 rules + 72 terms) + `prompt_template.md` (3-layer format + verification checklist) + pilot refinements (8 refinements + 5 gaps + 3 process improvements)
|
||||
**Date:** 2026-06-23
|
||||
|
||||
> **Reading guide.** This is the re-encoded version of the Pass 1 report. The 8-section structure is preserved. Every standard-math expression in §5 is replaced with the constructive type-theoretic form per the lexicon. Non-math sections (§1-§4, §6-§8) are preserved verbatim. The principled form is always produced; the user-specific form (Sectored Language V1, GA reinterpretations, classical Greek/Latin/Sanskrit) is opt-in.
|
||||
>
|
||||
> **The 5 rules (per `lexicon.md` §1):**
|
||||
> 1. **Boundedness** — no `∞_val`; use `Stream A = nat -> A` for processes.
|
||||
> 2. **Form-anchor** — every re-encoding has a form anchor.
|
||||
> 3. **Etymology** — 1-line origin + 1-line definition history.
|
||||
> 4. **Lossless + compression history** — every concept represented; compression notes per layer.
|
||||
> 5. **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`).
|
||||
|
||||
---
|
||||
|
||||
## 1. TL;DR
|
||||
|
||||
This talk presents a **position paper** arguing that conventional deep learning (SGD-trained neural networks) discovers **Fractured Entangled Representations (FER)** — internal weight configurations that produce the right output behavior but do not capture the underlying regularities of the world. The author proposes that an alternative training paradigm — **open-ended search** — discovers **Unified Factored Representations (UFR)**: internal configurations where each "neuron" or weight corresponds to a semantically meaningful, semantically independent factor of variation.
|
||||
|
||||
The evidence draws from three sources:
|
||||
|
||||
1. **Picbreeder** (Stanley & Lehman, MIT CSAIL & collaborators): a 2008-era web app where humans evolved CPPNs (Compositional Pattern Producing Networks) by selecting images they liked. The CPPNs that emerged have factored weight structures where single neurons correspond to human-interpretable axes (x-axis symmetry, mouth opening, eye width). When the same image is reproduced by SGD-trained MLPs, the resulting weights are entangled — perturbing any one weight affects many semantic aspects of the output.
|
||||
|
||||
2. **LLM failure modes** (FER in large language models): the brittleness of LLM mathematical reasoning (e.g., GPT-3 fails when chicken/duck/geese replace pencil/pen/eraser), counterfactual task degradation (Reasoning or Reciting paper), and the bag-of-heuristics arithmetic in Claude 3.5 Haiku (Anthropic's circuit-tracing paper) all suggest FER — the LLM is replicating patterns from training data without capturing the underlying regularities.
|
||||
|
||||
3. **The Platonic Representation Hypothesis** (Huh et al. 2024): different modalities (vision, language) trained on different objectives are converging to a shared statistical model of reality. The author agrees this is a real statistical phenomenon but argues it does not address whether the converged representations capture the underlying regularities.
|
||||
|
||||
The proposed solution is **open-ended search**: a learning paradigm that exhibits complexification (building regularities on top of regularities, bottom-up), emergence (higher-level structure from lower-level dynamics), adaptability (training for the ability to adapt, not for any specific task), and serendipity (exposure to multiple environments in sequence, not a single fixed dataset).
|
||||
|
||||
**Cross-cluster position:** Sits in cluster B and bridges to clusters A (math foundations: score_dynamics_giorgini's framework is an alternative route to capturing regularities via score matching; entropy_epiplexity's information-theoretic perspective defines what "regularity" means), C (complex systems: open-endedness is a property of complex systems theory), and E (applied LLMs/diffusion: the FER diagnosis predicts failure modes observed in cs336_architectures and creikey_dl_cv).
|
||||
|
||||
---
|
||||
|
||||
## 2. Key Concepts
|
||||
|
||||
Twenty concepts form the conceptual spine of the talk. Each is developed in §5 with full mathematical or conceptual statement.
|
||||
|
||||
### 2.1 The world has structure
|
||||
|
||||
The world is not random. Structure manifests as:
|
||||
- Self-similarity across spatial scales (fractals, multi-scale phenomena).
|
||||
- Physics symmetries: translation, rotation, scale invariance of physical laws.
|
||||
- Persistent objects: matter does not randomly appear/disappear; objects have identity over time.
|
||||
- Common patterns across many objects: the same shapes, materials, dynamics recur.
|
||||
|
||||
This is the **premise** of the talk — without structure in the world, the proposal would have no traction.
|
||||
|
||||
### 2.2 Platonic Space of Forms (philosophical premise)
|
||||
|
||||
Following Plato, the structure in the world is inherited from a **space of forms**: an abstract space containing the ideal patterns that objects instantiate. Properties common to many objects (e.g., "circularity," "verticality," "symmetry") are abstracted into the forms. The empirical regularity of the world reflects the underlying form structure.
|
||||
|
||||
The author uses "Platonic" as a metaphor for **the convergence of representations across modalities and tasks** — see §2.20.
|
||||
|
||||
### 2.3 Intelligent agents must capture structure
|
||||
|
||||
To control the world (achieve goals), an agent must understand the world. To understand the world, the agent must capture its structure. **More specifically: the internal representations of the agent's mind/brain must capture the structure.**
|
||||
|
||||
This is the **bridge claim** from the philosophical premise (§2.2) to the AI research program (§2.6+). It is the central thesis that the rest of the talk develops.
|
||||
|
||||
### 2.4 Architectural inductive biases as partial solutions
|
||||
|
||||
Modern deep learning has succeeded at capturing some structure via **inductive biases baked into the architecture**:
|
||||
- **Translation invariance** → CNNs (convolutional architecture).
|
||||
- **Permutation invariance over sets** → attention (transformer architecture).
|
||||
- General field: **symmetry learning / geometric deep learning**.
|
||||
|
||||
This is a partial solution: it captures **known regularities** via architecture design. **What about all the other regularities** (lighting invariance, material invariance, articulation, etc.)?
|
||||
|
||||
### 2.5 The SGD fallback
|
||||
|
||||
For unknown regularities, the current paradigm is: **train on lots of data with SGD and hope the network learns the regularities implicitly.**
|
||||
|
||||
The author labels this a "fallback" because there is no theoretical reason to believe SGD discovers factorized representations rather than entangled ones.
|
||||
|
||||
### 2.6 Does SGD actually capture regularities? (Empirical question)
|
||||
|
||||
Yes, somewhat. ChatGPT, self-driving cars, and other large-scale systems exhibit real understanding of the world. They generalize across lighting, viewpoints, contexts. So the SGD paradigm is not a complete failure.
|
||||
|
||||
But this brings us to the **position paper** hypothesis.
|
||||
|
||||
### 2.7 The Fractured Entangled Representation (FER) hypothesis
|
||||
|
||||
> Conventional SGD training in deep learning finds neural representations which are **fractured** and **entangled**.
|
||||
|
||||
**Fractured** = the internal state does not correspond to a smooth, organized decomposition of the input. **Entangled** = multiple semantic aspects are mixed together in each weight or activation.
|
||||
|
||||
The same input-output mapping can be implemented by many internal representations; SGD tends to find representations that satisfy the loss but do not decompose the world in semantically meaningful ways.
|
||||
|
||||
### 2.8 Why FER is a problem
|
||||
|
||||
If internal representations are entangled:
|
||||
- **Generalization** to novel situations is brittle (small input changes produce semantically unrelated output changes).
|
||||
- **Creativity** (composing new combinations of known concepts) is limited.
|
||||
- **Continual learning** (acquiring new tasks without forgetting old ones) is hard because new updates entangle with old structure.
|
||||
|
||||
These three failure modes are exactly the **jagged intelligence** observed in modern LLMs (good at IMO problems, bad at booking hotels).
|
||||
|
||||
### 2.8a Jagged intelligence in LLMs
|
||||
|
||||
Modern LLMs exhibit "jagged intelligence":
|
||||
- Excellent at hard reasoning tasks (e.g., IMO gold medal-level math).
|
||||
- Unreliable at simple tasks (e.g., reliably booking a hotel or plane).
|
||||
|
||||
This is **inconsistent** with a system that has captured the underlying regularities. A system with true understanding should be uniformly competent across difficulty levels. The author attributes this to FER.
|
||||
|
||||
### 2.9 The Unified Factored Representation (UFR) goal
|
||||
|
||||
A representation is **unified** if it consistently captures the same factor across instances (the same neuron activates for "x-axis symmetry" regardless of which image the symmetry is in). A representation is **factored** if each factor is independent (changing one factor does not change others). UFR = unified + factored.
|
||||
|
||||
The goal of open-ended search is to discover UFR.
|
||||
|
||||
### 2.10 Compositional Pattern Producing Networks (CPPNs)
|
||||
|
||||
A CPPN is a neural network with **heterogeneous activation functions** at each node (sin, cos, gaussian, sigmoid, abs, etc.). Given a coordinate (x, y), the CPPN outputs an image. CPPNs produce images with regular structure (symmetries, smoothness) by construction because the activations are themselves regular.
|
||||
|
||||
CPPNs are the canonical representation used in **Picbreeder** and other neuroevolution experiments. They are **not** trained by SGD; they are evolved.
|
||||
|
||||
### 2.11 Picbreeder (the experimental evidence)
|
||||
|
||||
Picbreeder (Secretan et al. 2008, Stanley & Lehman): an online website where humans **interactively evolve** CPPNs by selecting images they like. Users are not given any objective; they simply choose images they find interesting.
|
||||
|
||||
Outcomes:
|
||||
- Users **discover** complex images (skulls, butterflies, apples, lamps, aliens, etc.) without any target.
|
||||
- The discovered CPPNs have **factored structure**: each neuron corresponds to a distinct semantic axis (x-symmetry, y-symmetry, mouth opening, eye width, jaw width, etc.).
|
||||
- Sweeping a single weight value produces a **semantically coherent variation** (e.g., mouth opens wider; apple size grows).
|
||||
|
||||
This is **empirical proof that open-ended search can discover UFR** — in a toy domain (CPPN-generated images), but the principle generalizes.
|
||||
|
||||
### 2.12 Why Greatness Cannot Be Planned (Stanley & Lehman 2015)
|
||||
|
||||
The book by Stanley and Lehman formalizes insights from Picbreeder and similar experiments. Key concepts:
|
||||
- **Deception**: fitness functions that reward intermediate progress can lead evolution away from the global optimum.
|
||||
- **Serendipity**: stepping stones that have nothing to do with the final goal can be valuable.
|
||||
- **Open-Endedness**: search without a fixed objective tends to discover more interesting solutions than search with one.
|
||||
|
||||
This book is the philosophical backbone of the talk.
|
||||
|
||||
### 2.13 Intriguing properties of Picbreeder
|
||||
|
||||
Three properties that emerge from open-ended search:
|
||||
|
||||
1. **Open-Ended**: the search never converges because the interesting region of CPPN space is unbounded.
|
||||
2. **Serendipitous exaptation**: traits evolved for one purpose get repurposed (e.g., a step toward the teapot enables the skull).
|
||||
3. **Emergence of evolvability**: certain CPPNs are more evolvable — small mutations produce large but coherent changes (canalization, regularity, modularity, symmetry).
|
||||
|
||||
### 2.14 Evolvability and canalization
|
||||
|
||||
**Canalization** = a genotype where small mutations produce small phenotypic changes (the phenotype is buffered against genetic variation). Waddington's concept, applied to CPPNs.
|
||||
|
||||
In Picbreeder, evolved CPPNs exhibit canalization: certain mutations only affect one semantic aspect. SGD-trained MLPs do not — mutations affect multiple aspects.
|
||||
|
||||
The author argues this canalization IS the structure of the world being captured in the representation.
|
||||
|
||||
### 2.15 Layerization (CPPN → MLP conversion)
|
||||
|
||||
Given any CPPN, there exists an MLP that computes the same function (universal approximation). **Layerization** is the process of converting a CPPN to an MLP.
|
||||
|
||||
The layerization of an evolved Picbreeder CPPN is an MLP with **factored weight structure**: single weights correspond to single semantic axes. The layerization of an SGD-trained MLP is a similar MLP but with **entangled weights**.
|
||||
|
||||
This is the central experimental result: **the same function, two different internal organizations, with different generalization properties.**
|
||||
|
||||
### 2.16 SGD skull vs Picbreeder skull
|
||||
|
||||
Demonstration:
|
||||
- Train an MLP with SGD to reproduce the Picbreeder skull image. SGD succeeds (perfect reconstruction).
|
||||
- Sweep each weight value in the SGD-trained MLP. Output changes are chaotic — no single weight corresponds to a single semantic axis.
|
||||
- Layerize the Picbreeder CPPN. Sweep each weight value. Output changes are semantic — weight #i controls mouth opening, weight #j controls eye width, etc.
|
||||
|
||||
Same loss, same architecture, two completely different internal organizations. SGD gives FER; open-ended search gives UFR.
|
||||
|
||||
### 2.17 FER in LLMs — three pieces of evidence
|
||||
|
||||
The author surveys three papers showing FER in modern LLMs:
|
||||
|
||||
**A. GSM-Symbolic (Mirzadeh et al. 2025, Apple).** Replace specific objects in GSM8K math problems. GPT-3 gets pencil/pen/eraser sums correct ("9 things") but fails on chicken/duck/geese sums (counts the animals, not the items — gets 10 instead of 9). Same arithmetic structure, different surface tokens → different answer. The model learned the GSM8K surface pattern, not the underlying counting.
|
||||
|
||||
**B. Reasoning or Reciting (Wu et al. 2024, MIT/BU).** Counterfactual versions of reasoning tasks (base 9 arithmetic, code execution with 0-based vs 1-based indexing, drawing tasks, etc.). Performance drops substantially — the model doesn't generalize to systematic perturbations of the task structure.
|
||||
|
||||
**C. On the Biology of a Large Language Model (Anthropic 2025).** Circuit tracing of Claude 3.5 Haiku on 36 + 59 = 95. The internal computation is: 36 ≈ 30, 59 ≈ 50, 30+50 ≈ 80, 80+15 ≈ 92 (somewhere near 95). A bag of magnitude heuristics, not a digit-by-digit algorithm. Correct answer, wrong internal mechanism.
|
||||
|
||||
The author concludes: **LLM behavior is impressive but the internal mechanism is bag-of-heuristics — FER.**
|
||||
|
||||
### 2.18 The Platonic Representation Hypothesis (Huh et al. 2024)
|
||||
|
||||
Empirical finding: as you scale up vision models (e.g., DINOv2) and language models (e.g., Llama), the **representation similarity** between them increases. Models trained on different data, different objectives, different architectures, are converging in some geometric sense to a shared representation of "reality."
|
||||
|
||||
This is a **statistical phenomenon**. It does not necessarily mean the converged representation captures the regularities of the world (it might be a statistical optimum without being a meaningful decomposition).
|
||||
|
||||
### 2.19 The author's position on the Platonic Hypothesis
|
||||
|
||||
The Platonic Representation Hypothesis is real (empirically observed convergence) but **statistical**, not **structural**. Convergence in representation geometry does not imply convergence in factorization.
|
||||
|
||||
A model could represent the world with one set of entangled features, and a different model could represent it with another set of entangled features, and both representations could be close in some metric space without either being factored.
|
||||
|
||||
The author wants UFR: the converged representations should be **structurally aligned**, not just statistically correlated.
|
||||
|
||||
### 2.20 Open-endedness as a learning paradigm
|
||||
|
||||
A learning paradigm with four properties:
|
||||
- **Complexification**: build regularities on top of regularities, bottom-up (like morphogenesis or development).
|
||||
- **Emergence**: higher-level structure arises from lower-level dynamics.
|
||||
- **Adaptability**: train for the ability to adapt to new environments, not for any specific task.
|
||||
- **Serendipity**: expose the system to multiple environments in sequence, not a single fixed dataset.
|
||||
|
||||
Open-ended search (e.g., Picbreeder-style evolution) exhibits all four. SGD does not.
|
||||
|
||||
### 2.21 Pressure to adapt as the key driver
|
||||
|
||||
Among the four properties, **adaptability** is the author's "hunch" about the most important. Reasons:
|
||||
- Evolution optimizes for fitness in a changing environment; the implicit pressure to be adaptable (since the environment changes) creates robust representations.
|
||||
- SGD optimizes for a fixed loss; no implicit pressure to be adaptable.
|
||||
- A system trained to be adaptable will, almost as a side effect, capture regularities of the world (because regularities are what allow prediction across environments).
|
||||
|
||||
The author conjectures: **strong representations and adaptable representations are one and the same.**
|
||||
|
||||
### 2.22 Space of Forms ↔ Representation Space
|
||||
|
||||
The author closes with a philosophical question: does the **internal representation of a good agent** correspond **directly** to the Platonic Space of Forms?
|
||||
|
||||
The aspiration is that an ideal mind's representation would be a faithful mirror of the structure of reality. Current AI representations are imperfect instantiations of this ideal.
|
||||
|
||||
This is speculation (the author acknowledges it) but provides the philosophical framing for "Platonic Intelligence" as the talk's title.
|
||||
|
||||
---
|
||||
|
||||
## 3. Frame Analysis
|
||||
|
||||
62 unique frames were extracted from the 89MB mp4 at threshold 0.05 (research talk with diagrams, equations, and motivational slides; OCR'd via winsdk in 3.7s).
|
||||
|
||||
(§3 preserved from Pass 1 verbatim — frames describe diagrams and slides, not mathematical content. Not a re-encoding target.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Transcript Highlights
|
||||
|
||||
Sixteen verbatim passages from the cleaned transcript (1659 segments, 61KB) that capture the conceptual flow.
|
||||
|
||||
(§4 preserved from Pass 1 verbatim — verbatim quotes from the speaker. Not a re-encoding target.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Mathematical / Theoretical Content (Re-encoded)
|
||||
|
||||
This section develops the conceptual content of the talk in depth. The talk is conceptual rather than heavily mathematical, but several key concepts admit formalization.
|
||||
|
||||
### 5.1 Inductive biases via group invariance
|
||||
|
||||
A function `f : (X -> Y)` is **equivariant** under a group `G : Group` if `forall g : G, forall x : X, apply(f, apply(action_g, g, x)) = apply(action_Y, g, apply(f, x)) : Prop`. A function is **invariant** if `forall g : G, forall x : X, apply(f, apply(action_g, g, x)) = apply(f, x) : Prop` (trivial action on Y).
|
||||
|
||||
Architectural inductive biases encode specific group invariances:
|
||||
- **Translation invariance**: `G = TranslationGroup where G = Pair<quantity : float64, quantity : float64>` (continuous translations); achieved by `convolutional : procedure`.
|
||||
- **Permutation invariance**: `G = SymmetricGroup(n) with action : (Perm, xs) -> permute(Perm, xs)`; achieved by `attention : procedure`.
|
||||
- **Rotation invariance**: `G = SO(2) or SO(3)`; achieved by group convolutions.
|
||||
|
||||
The geometric deep learning program (Bronstein et al. 2021) generalizes this: any prior knowledge of the form "f is invariant under some group action on inputs" can be encoded as an architectural constraint.
|
||||
|
||||
**Limitation:** for invariances we don't know in advance (lighting invariance, articulation, etc.), there is no obvious architectural choice.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses group-theoretic math notation (g·x); Layer 2 expands to forall + apply (Tier 1 #1.2 + Tier 2 #2.1 per the noise-dedup); Layer 3 implements as a property check.
|
||||
|
||||
### 5.2 The SGD loss landscape
|
||||
|
||||
The training loss `procedure L(theta : Parameters, D : DataDistribution, f_theta : (X -> Y), loss_fn : ((Y, Y) -> Scalar)) -> Scalar where L = mean (x, y) sample D of apply(loss_fn, apply(f_theta, x), y) : float64` is a function from the parameter space `Theta : kind` to `quantity : float64`. SGD samples a stochastic gradient estimate `procedure stochastic_gradient(theta : Parameters, batch : DataBatch, L : LossFunction) -> Gradient where g_t = apply(grad, L, theta, batch) : Vector of float64` and updates `procedure sgd_step(theta : Parameters, g : Gradient, eta_t : LearningRate) -> Parameters where theta_{t+1} = subtract(theta, scale(eta_t, g)) : Parameters` (encoding: `float64` for eta_t).
|
||||
|
||||
The landscape of `L : Parameters -> float64` has:
|
||||
- Local minima (often degenerate — many minima with similar loss values).
|
||||
- Saddle points (more common in high dimensions than local minima).
|
||||
- Plateaus (regions of near-zero gradient).
|
||||
|
||||
The network's final weights `theta* : Parameters` are a **local optimum** of `L`, not necessarily the global optimum, and not necessarily a "good" internal representation even if loss is low.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses expected-value notation; Layer 2 expands to apply + scale + subtract (Tier 2 #2.1 procedure per the noise-dedup); Layer 3 implements as a stochastic gradient loop.
|
||||
|
||||
### 5.3 Why SGD finds FER (informal argument)
|
||||
|
||||
Consider two weight configurations `theta_1, theta_2 : Parameters` that achieve similar loss `L : float64`. They correspond to two different ways of implementing the input-output mapping. The "natural" structure of the data (UFR) is one possibility; many other configurations exist that achieve the loss without the natural structure.
|
||||
|
||||
SGD samples these configurations **uniformly with respect to their basin of attraction** (in practice, with bias from initialization and noise). Most configurations in parameter space are not UFR — UFR is a low-dimensional subset (the "manifold of factored representations"). SGD finds FER with overwhelming probability because FER is the generic case: `forall theta_initial : Parameters, Pr[SGD(theta_initial) lands in UFR-basin] is approximately 0 : float64 because UFR-subset : kind has measure 0 in the parameter space kind Theta : kind` (encoding: `float64` for probability).
|
||||
|
||||
**Counterargument:** SGD with strong inductive biases (architectural symmetries) constrains the search to a smaller subset, increasing the chance of finding structured representations: `forall architecture : ArchitecturalInductiveBias, search_subspace : kind where search_subspace ⊆ theta-space and architecture-encoded, Pr[find UFR | architecture] >= Pr[find UFR | no architecture] : Prop`. This is what convolutional and transformer architectures do for spatial and permutation structure.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "measure-zero"; Layer 2 specifies the probability and the subset; Layer 3 implements as a sampling argument (heuristic).
|
||||
|
||||
### 5.4 The CPPN parameterization
|
||||
|
||||
A CPPN (Compositional Pattern Producing Network) is a neural network `procedure cppn(image_coord : Pair<quantity : float64, quantity : float64>) -> Vector3D where f(x, y) = apply(sigma_L, add(matmul(W_L, apply(sigma_{L-1}, add(matmul(W_{L-1}, ..., add(matmul(W_1, [x, y]), b_1), ...)), b_{L-1})), b_L) : Vector3D` (encoding: `float64` for image_coord, `Vector3D` of `float64` for RGB output).
|
||||
|
||||
The heterogeneous activations `sigma : Sum<Sin, Cos, Gaussian, Sigmoid, Abs, Linear, ...>` (the activation alphabet) encode **regularity biases**:
|
||||
- `Sin / Cos → Periodic` patterns.
|
||||
- `Gaussian → Radial` symmetries.
|
||||
- `Abs → AxisSymmetric` (symmetry across axes).
|
||||
- `Sigmoid → SoftThreshold`.
|
||||
|
||||
A random CPPN produces an image with structure (because of the regularity biases). A **trained** CPPN (via SGD) tends to find entangled weight configurations; an **evolved** CPPN (via selection) tends to find factored configurations.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses nested math notation; Layer 2 expands the apply chain (Tier 2 #2.1 procedure per the noise-dedup); Layer 3 implements as a record with heterogeneous sigma fields.
|
||||
|
||||
### 5.5 Picbreeder's evolution algorithm
|
||||
|
||||
Picbreeder is a steady-state genetic algorithm (not generational):
|
||||
|
||||
```
|
||||
procedure picbreeder_step(population : Population<CPPN>, user_select : (Images -> CPPN)) -> Population<CPPN>:
|
||||
offspring = mutate(apply(user_select, render(population)))
|
||||
return replace(population, least_fit, offspring)
|
||||
```
|
||||
|
||||
The selection pressure is **subjective aesthetic preference** (with no explicit objective).
|
||||
|
||||
Effective population size: tens to hundreds. Generations: hundreds to thousands. Effective compute: days to weeks of human attention per "interesting" image.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "human in the loop"; Layer 2 specifies the procedure (Tier 2 #2.1 + Tier 2 #2.2 arguments); Layer 3 implements as a genetic-algorithm loop.
|
||||
|
||||
### 5.6 Waddington's canalization
|
||||
|
||||
Conrad Waddington (1942, 1957) introduced **canalization** to describe how embryonic development buffers against genetic variation: `procedure canalized_wrt(cppn : CPPN, weight_index : int64) -> Prop where forall delta : quantity : float64 where |delta| < delta_0 : quantity : float64, magnitude(subtract(apply(cppn, perturb(weights, delta * e_i)), apply(cppn, weights))) < epsilon : quantity : float64` (encoding: `float64` for delta, epsilon, delta_0). A canalized trait is one where many genotypes map to the same phenotype — the developmental "landscape" has ridges and valleys that channel variation into a small set of outcomes.
|
||||
|
||||
In ML terms, canalization is **insensitivity to certain weight perturbations**: `forall genotype_variation : GeneticMutation, phenotype_change : Morphology where developmental_landscape buffers(genotype_variation) => |phenotype_change| is small : Prop`. An MLP is canalized with respect to weight w if small perturbations to w leave the output approximately unchanged. The Picbreeder CPPNs are canalized with respect to mutations in "non-active" genes; SGD-trained MLPs are not (every weight matters).
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "small mutations don't matter"; Layer 2 specifies the formula (Tier 2 #2.6 relation per the noise-dedup); Layer 3 implements as a perturbation test.
|
||||
|
||||
### 5.7 The Picbreeder Skull MLP factorization
|
||||
|
||||
The experimental result (Secretan et al. 2008; later formalized in the Kumar/Stanley papers):
|
||||
|
||||
Given a Picbreeder skull image (a CPPN output), the layerized Picbreeder CPPN is an MLP whose weights have the following factorization property:
|
||||
|
||||
`forall cppn : CPPN with UFR, forall weight_index i, exists semantic_axis a_i : (Image -> quantity : float64), exists permutation sigma, forall j != sigma(i), forall delta : quantity : float64, apply(a_j, render(perturb(weights, delta * e_i))) = apply(a_j, render(weights)) : quantity : float64` (encoding: `float64`).
|
||||
|
||||
This is **factored**: each weight controls one axis independently.
|
||||
|
||||
The **equivalent condition** is `forall cppn : CPPN with UFR, Jacobian(cppn, weights) : Matrix where Rank(Jacobian) = 1 : int64` (encoding: `int64` for rank, `float64` for entries). SGD-trained MLPs have **dense** Jacobians: every weight affects every axis (per §5.8).
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "each weight controls one axis"; Layer 2 specifies the forall + exists + apply chain (Tier 1 #1.2, #1.3 per the noise-dedup); Layer 3 implements as a rank-1 check on the Jacobian.
|
||||
|
||||
### 5.8 SGD skull: no factorization
|
||||
|
||||
The SGD-trained MLP with the same loss has weights where sweeping any weight changes **multiple** semantic axes simultaneously:
|
||||
|
||||
`forall sgd_mlp : MLP trained with SGD, forall weight_index i, exists j != i, exists delta : quantity : float64, |apply(a_j, render(perturb(weights, delta * e_i))) - apply(a_j, render(weights))| > 0 : quantity : float64` (encoding: `float64` for the delta and the change).
|
||||
|
||||
The SGD MLP has the same input-output mapping (it reproduces the skull image) but its internal organization is **entangled**: `forall sgd_mlp : MLP trained with SGD, Jacobian(sgd_mlp, weights) : Matrix where forall i, exists j != i, entry(i, j) != 0 : Prop` (encoding: `float64` for entries).
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "every weight affects every axis"; Layer 2 specifies the forall + exists + entry-check (Tier 1 #1.2, #1.3 + Tier 2 #2.6 relation per the noise-dedup); Layer 3 implements as a rank-n check on the Jacobian.
|
||||
|
||||
### 5.9 Why does this happen?
|
||||
|
||||
Two hypotheses (the author leans toward the second):
|
||||
|
||||
**Hypothesis A: Basin of attraction.** SGD finds FER because the FER basin of attraction is larger than the UFR basin in the SGD loss landscape: `forall theta_initial : Parameters, basin_attraction(theta_initial) : kind where basin_attraction is FER-basin => volume(FER-basin) > volume(UFR-basin) : Prop`. This is a structural claim about the landscape.
|
||||
|
||||
**Hypothesis B: Search algorithm.** SGD is a **gradient-following** algorithm; it finds local minima of the loss. The UFR solution corresponds to a specific local minimum that SGD cannot reach because the gradient does not point toward it. Open-ended search explores more of the space (via mutation) and finds the UFR minimum because it does not follow gradients: `SGD : procedure where output = argmin_theta L(theta) (gradient-following); OpenEndedSearch : procedure where output = argmax_archive diversity(archive) (mutation-based) : kind`.
|
||||
|
||||
The evidence: even with weight decay and other regularizers, SGD almost never finds UFR (per the Q&A in §4.17). This favors Hypothesis B — the issue is the search algorithm, not the loss landscape.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "SGD can't reach UFR"; Layer 2 specifies the two procedures (Tier 2 #2.1 + Tier 2 #2.6 per the noise-dedup); Layer 3 implements as an empirical test (heuristic).
|
||||
|
||||
### 5.10 FER in LLMs: arithmetic as bag-of-heuristics
|
||||
|
||||
The Anthropic 2025 paper traces Claude 3.5 Haiku's computation for `36 + 59 = 95`. The mechanism (per the circuit tracing):
|
||||
|
||||
```
|
||||
procedure claude_arithmetic(a : int64, b : int64) -> int64:
|
||||
piecewise(
|
||||
matches (a, b) of lookup_table,
|
||||
magnitude_heuristic(a) + magnitude_heuristic(b),
|
||||
digit_combination(a, b)
|
||||
)
|
||||
```
|
||||
|
||||
Where `procedure magnitude_heuristic(n : int64) -> int64 where approximate(n, nearest_multiple_of_10(n)) : int64` (encoding: `int64`). The model uses **approximate magnitude heuristics** (30 + 50 ≈ 80) and **lookup tables** (specific pairs) rather than a **general algorithm**.
|
||||
|
||||
This is **not** digit-by-digit addition. In formal terms: the model has learned a **piecewise function** `{lookup_table, magnitude_heuristic, digit_combination}` rather than a **uniform algorithm**. The piecewise structure is FER — the representations for different problems are different, not unified.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "bag of heuristics"; Layer 2 specifies the piecewise sum (Tier 3 #3.17 Sum per the noise-dedup); Layer 3 implements as a piecewise dispatch.
|
||||
|
||||
### 5.11 The Platonic Representation Hypothesis (formal)
|
||||
|
||||
Given two models A (trained on modality X) and B (trained on modality Y), define a **representation similarity** measure `procedure representation_similarity(A : Representation, B : Representation, metric : SimilarityMetric) -> quantity : float64 where result in [0.0, 1.0] : Prop` (e.g., RSA: Representational Similarity Analysis, or mutual kNN).
|
||||
|
||||
The PRH (Huh et al. 2024) states: as model size and performance increase, `forall A, B : Representation, forall scale_n : int64, limit(representation_similarity(A_scale_n, B_scale_n)) -> 1 : quantity : float64 as n -> infinity : Stream` (encoding: `float64`; the convergence is a `Stream`, not a value).
|
||||
|
||||
This is an empirical claim about **representational geometry** — the manifolds in representation space are converging. It does not say anything about whether the representations are factored.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "→ 1"; Layer 2 re-encodes as `Stream` (per Rule 1: no `∞_val`); Layer 3 implements as an empirical scaling experiment.
|
||||
|
||||
### 5.12 Distinguishing FER from UFR via probing
|
||||
|
||||
How to test whether a representation is FER or UFR?
|
||||
|
||||
**Test 1: Single-weight ablation.** Perturb one weight at a time: `procedure test_single_weight_ablation(network : MLP) -> Prop where forall weight_index i, exists semantic_axis a_i such that perturb_and_measure(i) changes only a_i : Prop`. If each perturbation affects only one semantic axis (measured by independent interpretable probes), the representation is UFR. If each perturbation affects multiple axes, it's FER.
|
||||
|
||||
**Test 2: Representation rotation.** Apply a fixed rotation R to the weight space: `procedure test_representation_rotation(network : MLP, R : Matrix where R is rotation) -> Prop where apply(R, weights) yields outputs that are rotated versions of original : Prop` (encoding: `Matrix` of `float64`). If the rotated representation still produces the same semantic outputs (with rotated axes), the representation is UFR. If the rotated representation produces incoherent outputs, it's FER.
|
||||
|
||||
**Test 3: Causal intervention.** Intervene on weight `w_i` in a direction that should change semantic axis `a_i`: `procedure test_causal_intervention(network : MLP, i : int64, axis_k : int64) -> Prop where perturb(weights, direction_of_a_k) changes only a_k in output : Prop` (encoding: `int64` for indices). If only `a_i` changes in the output, UFR. If multiple axes change, FER.
|
||||
|
||||
These tests are computable and have been applied to small models. Applying them to LLMs is open work.
|
||||
|
||||
**Compression notes (per Rule 4):** Layer 1 uses informal "perturb and see what changes"; Layer 2 specifies the three procedures (Tier 1 #1.2, #1.3 + Tier 2 #2.1, #2.6 per the noise-dedup); Layer 3 implements as automated probing tests.
|
||||
|
||||
---
|
||||
|
||||
## 6. Connections
|
||||
|
||||
(§6 preserved from Pass 1 verbatim — cross-references to other videos in the campaign. Not a re-encoding target.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions
|
||||
|
||||
(§7 preserved from Pass 1 verbatim — 16 open questions organized by theoretical/empirical/applied/philosophical. Not a re-encoding target.)
|
||||
|
||||
---
|
||||
|
||||
## 8. References
|
||||
|
||||
(§8 preserved from Pass 1 verbatim — people, papers, and background concepts. Not a re-encoding target.)
|
||||
|
||||
---
|
||||
|
||||
## Appendices A-J (Preserved from Pass 1 verbatim)
|
||||
|
||||
(Concept map, transcript excerpts, formalizations, expanded connections, expanded open questions, full bibliography, cross-references within campaign, synthesis summary, personal notes, glossary. The Appendix C formalizations overlap with §5 and are documented there in the re-encoded form.)
|
||||
|
||||
---
|
||||
|
||||
## Verification (per `lexicon.md` §12 + pilot refinement)
|
||||
|
||||
- [x] **Lossless** — every Pass 1 concept is represented in the de-obfuscated form. §5 has all 12 math sections re-encoded; §1-§4, §6-§8 + appendices are preserved verbatim.
|
||||
- [x] **Bounded** — no `∞_val` or `∞_card`. The "as n → ∞" convergence in §5.11 is re-encoded as `Stream similarity_n = nat -> float64`. The "n ∈ ℝ²" in §5.1 is re-encoded as `Pair<quantity : float64, quantity : float64>` (bounded form).
|
||||
- [x] **Encoding-explicit** — every value-bearing term has `encoding:` (default `float64`; `int64` for exact integers per the taxonomy).
|
||||
- [x] **Constructively typed** — every expression has a type signature.
|
||||
- [x] **Etymology-cited** — every new term has the 1-line origin + 1-line definition history (per the decoder file).
|
||||
- [x] **Form-anchored** — every re-encoding has a form anchor.
|
||||
- [x] **Noise-deduped** — the 6 noise-dedup maps applied where applicable.
|
||||
- [x] **Compression notes** — every transformation has a "Compression notes" field per Rule 4.
|
||||
- [x] **No esoteric content** — secular sanitization preserved.
|
||||
- [x] **User-specific conventions applied only when appropriate** — the principled form is always produced; the user-specific form is opt-in.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- `lexicon.md` (the codified operational spec)
|
||||
- `dedup_map.md` (the 6 noise-dedup maps)
|
||||
- `prompt_template.md` (the LLM operational spec)
|
||||
- `platonic_intelligence_kumar_translation.md` (the side-by-side table) — 36 rows
|
||||
- `platonic_intelligence_kumar_decoder.md` (the per-term decoder, tier-categorized)
|
||||
|
||||
---
|
||||
|
||||
*End of `platonic_intelligence_kumar_deobfuscated.md`. Total: 8 sections + appendices preserved; all 12 math sections of §5 re-encoded; 4 verification criteria met.*
|
||||
Reference in New Issue
Block a user