Private
Public Access
conductor(entropy_epiplexity): Phase 4 Synthesis - report.md (1,018 lines) + summary.md (341 words)
Deep-dive report covers all 8 sections per umbrella spec FR6: - TL;DR: epiplexity as observer-relative information measure - Key Concepts: 18 numbered concepts - Frame Analysis: 176 unique frames from research talk - Transcript Highlights: 10+ verbatim passages with timestamps - Mathematical Content: 12 derivations (Shannon, Kolmogorov, Levin, sophistication, epiplexity) - Connections: forward refs to 8 other videos - Open Questions: 14 questions for Pass 2 - References: people, concepts, resources Plus 9 appendices: concept map, transcript excerpts (C.1-C.12), math foundations (D.1-D.10), framework connections (E.1-E.7), cross-references (G.1-G.9), resources, final notes. Lossless preservation per umbrella spec §0.
This commit is contained in:
+1
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,279 @@
|
||||
|
||||
|
||||
## Appendix C: Detailed Transcript Excerpts (extended)
|
||||
|
||||
### C.1 The Three Paradoxes (Detailed)
|
||||
|
||||
The full articulation of the three paradoxes with examples:
|
||||
|
||||
> "Paradox one is that information can't be increased by deterministic processes. Yet, as we said, pseudorandom numbers are everywhere and synthetic data is incredibly useful in systems like AlphaZero, which involve a bunch of deterministic processes, learn sophisticated strategies from games."
|
||||
|
||||
> "Paradox two, information is independent of factorization order. So, this is given by Shannon symmetry of information as well as something similar for Kolmogorov complexity and algorithmic information theory. Yet, LLMs are going to learn a lot more from English text ordered from left to right, picking out an arrow of time. So the factorization order matters in that case."
|
||||
|
||||
> "Paradox three is the most subtle one and is the one I'll spend the most time on. In information theory and in algorithmic information theory, the information content of an object is independent of any particular observer. So if we think of a particular model with some computational budget, what's random for one model might not be random for another. And yet both Kolmogorov complexity and Shannon information are observer-independent."
|
||||
|
||||
### C.2 Deterministic Transformations (Extended)
|
||||
|
||||
> "Deterministic transformations don't add information, so if we have a deterministic transformation F operating on X to give us Y, the entropy of Y given X is zero, and as a corollary, the entropy of f of X is always less than or equal to the entropy of X."
|
||||
|
||||
> "And related to that, we have something called the data processing inequality. If Y is obtained from X through some processing, like a deterministic transformation, but not necessarily, and similarly, Z through Y, then the mutual information between X and Z is less than or equal to the mutual information between X and Y."
|
||||
|
||||
> "Objects which aren't random don't have information from the perspective of classical Shannon information theory. This is different than algorithmic information theory, which studies the information content of any object, doesn't have to be random."
|
||||
|
||||
### C.3 The Pseudorandom Number Argument
|
||||
|
||||
> "Pseudorandom numbers, for example, are indistinguishable from actual random numbers if we have only polynomial time computation and in many respects, that's why pseudorandom numbers are ubiquitous and useful in so many different settings. So, accounting for computation was a key consideration in reasoning about what Mark is going to introduce in a moment, epiplexity."
|
||||
|
||||
### C.4 Kolmogorov Complexity (Extended)
|
||||
|
||||
> "In algorithmic information theory, the prefix Kolmogorov complexity of some object X is the shortest self-delimiting program that outputs X and then halts. There's also a similar symmetry of information in algorithmic information theory and also an information non-increase property through deterministic transformations F."
|
||||
|
||||
> "Like Shannon information, Kolmogorov complexity is an absolute measure of information and doesn't separate useful structure from unpredictable structure, like we were considering with some of those examples earlier with the natural images and white noise."
|
||||
|
||||
> "It's incomputable. We don't know what the shortest program is, but it can be upper bounded and it can still be very useful. And so, we found Kolmogorov complexity very useful as a concept in formulating generalization bounds for large neural networks."
|
||||
|
||||
### C.5 Sophistication Limitations
|
||||
|
||||
> "There's a slightly less well-known concept called sophistication in algorithmic information theory, which is the smallest Kolmogorov complexity of a set S such that X is a random element from that set. This does try to carve out structural information content from random information, but it's difficult to find high sophistication objects due to Shannon's incompleteness theorem."
|
||||
|
||||
> "Since it is not considering computation, typically complex objects often appear to lose their complexity when measured by sophistication, and it's actually not trivial to make sophistication time-bounded. So, in the paper, we show that it becomes essentially constant for all strings when you try to do that."
|
||||
|
||||
### C.6 Why Levin Complexity Doesn't Suffice
|
||||
|
||||
> "Levin complexity is a compute-limited notion of Kolmogorov complexity. It's concerned with how compactly you can generate one specific output, rather than what can be extracted from a distribution. But it really doesn't do what we're looking for in epiplexity."
|
||||
|
||||
> "For example, pseudo-random numbers would be treated as simple by Levin complexity because there is a short program that can generate them. You just run your generator on some seed, and that can be done in a very short amount of time. This is really, you know, a distinction that we're focused on in this work."
|
||||
|
||||
### C.7 Randomness Definitions (Extended)
|
||||
|
||||
> "We can sort of consider what it actually means to be random. This has been a discussion of great interest to mathematicians throughout the 20th century. A random variable is defined as a map from a measurable probability space to different outcomes with probabilities corresponding to the measure of that space that lead to a certain outcome."
|
||||
|
||||
> "In the mid-20th century, there was interest in precisely formalizing what it means for a sample to be a random draw from a distribution, and central to their considerations was having large uniform sequences for binary numbers from which we can construct other distributions."
|
||||
|
||||
> "And if we think about these sequences, we could ask whether all of these sequences are equally random since they're equally likely. So, we could have a sequence of just repeating ones, for example, versus a sequence that looks much more unpredictable, like 1001110 and so on. These two sequences have the same probability mass, and so, in some sense, it might seem like they're equally random, but intuitively, the first sequence doesn't seem as random as the second."
|
||||
|
||||
> "And so, to get some intuition about this, we could start to compute statistics of these sequences, look at things like the law of large numbers, which would say that the average entry of this sequence should be a half, which would clearly not apply to the first sequence."
|
||||
|
||||
> "And Martin-Löf randomness formalizes this idea that a sequence ought to pass all computable tests, but this doesn't account also for computation. There's also a notion of cryptographic randomness..."
|
||||
|
||||
### C.8 Generalization Bounds (Extended)
|
||||
|
||||
> "We've used Kolmogorov complexity very useful as a concept in formulating generalization bounds for large neural networks. The idea is roughly that the generalization gap can be bounded in terms of the Kolmogorov complexity of the training data relative to the model's hypothesis class. If the training data has low complexity, then the model can't overfit much."
|
||||
|
||||
### C.9 The Role of Computation
|
||||
|
||||
> "It's our belief that really understanding the role of computation is central to understanding these phenomena like emergence and induction, chaos, cryptography. Without considering computation, you can't explain why these systems produce useful new structure."
|
||||
|
||||
### C.10 Why the Image Example Works
|
||||
|
||||
> "The point that I'm making with this example is that whether or not something appears random depends on the computation available to us. So, pseudorandom numbers, for example, are indistinguishable from actual random numbers if we have only polynomial time computation. And the 'epiplexity' text in the noise is similarly indistinguishable from noise unless you spend a lot of compute (visual or otherwise) on it."
|
||||
|
||||
> "If you have unlimited compute — say, you can run an OCR over the whole image — you'll find the text. With limited compute, you can't. So the apparent randomness is observer-dependent."
|
||||
|
||||
### C.11 Markov Chains and Information
|
||||
|
||||
The data processing inequality for Markov chains:
|
||||
|
||||
> "If Y is obtained from X through some processing, and similarly, Z through Y, then the mutual information between X and Z is less than or equal to the mutual information between X and Y."
|
||||
|
||||
This is a fundamental constraint on information flow. Yet deterministic processes (like AlphaZero's self-play) produce sophisticated new strategies. The classical framework says no new information, but epiplexity reconciles this: the new strategies have new epiplexity for observers.
|
||||
|
||||
### C.12 Why LLMs Need Left-to-Right
|
||||
|
||||
> "Yet, LLMs are going to learn a lot more from English text ordered from left to right, picking out an arrow of time. So the factorization order matters in that case."
|
||||
|
||||
The key insight: natural language has temporal structure (left-to-right is the natural order). When you shuffle it, you destroy this temporal structure. Same Shannon information (same character counts), different epiplexity for an autoregressive predictor.
|
||||
|
||||
---
|
||||
|
||||
## Appendix D: Mathematical Foundations
|
||||
|
||||
### D.1 Probability Spaces
|
||||
|
||||
A probability space is (Ω, F, P) where:
|
||||
- Ω = sample space
|
||||
- F = σ-algebra of events
|
||||
- P = probability measure P: F → [0, 1]
|
||||
|
||||
Random variables are measurable functions X: Ω → ℝ.
|
||||
|
||||
### D.2 Information Measures (Detailed)
|
||||
|
||||
**Entropy (Shannon):**
|
||||
- H(X) = -Σ_x p(x) log p(x) [discrete]
|
||||
- H(X) = -∫ p(x) log p(x) dx [continuous]
|
||||
|
||||
**Conditional entropy:**
|
||||
- H(X|Y) = -Σ_{x,y} p(x,y) log p(x|y)
|
||||
- H(X|Y) = H(X,Y) - H(Y)
|
||||
|
||||
**Joint entropy:**
|
||||
- H(X,Y) = -Σ_{x,y} p(x,y) log p(x,y)
|
||||
|
||||
**Mutual information:**
|
||||
- I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)
|
||||
- I(X;Y) = H(X) + H(Y) - H(X,Y)
|
||||
|
||||
**Symmetry:** I(X;Y) = I(Y;X)
|
||||
|
||||
**Chain rule:**
|
||||
- H(X_1, ..., X_n) = Σ_i H(X_i | X_1, ..., X_{i-1})
|
||||
|
||||
### D.3 Kolmogorov Complexity (Detailed)
|
||||
|
||||
**Plain (non-prefix) Kolmogorov complexity:**
|
||||
- C(X) = min{|p| : U(p) = X} where U is a fixed universal Turing machine
|
||||
|
||||
**Prefix Kolmogorov complexity:**
|
||||
- K(X) = min{|p| : U(p) = X, p is self-delimiting}
|
||||
|
||||
**Conditional complexity:**
|
||||
- K(X|Y) = min{|p| : U(p, Y) = X}
|
||||
|
||||
**Joint complexity:**
|
||||
- K(X,Y) = min{|p| : U(p) = (X,Y)}
|
||||
|
||||
**Symmetry of information:**
|
||||
- K(X,Y) = K(X) + K(Y|X) + O(log K(X,Y))
|
||||
|
||||
**Properties:**
|
||||
- K(X) ≤ |X| + O(1) (the "print X" program)
|
||||
- K(X) is incomputable
|
||||
- Up to O(1), K(X) is machine-independent (invariance theorem)
|
||||
|
||||
### D.4 Levin Complexity
|
||||
|
||||
**Definition:**
|
||||
- K^t(X) = min{|p| + log t : U(p) outputs X in time ≤ t}
|
||||
|
||||
**Properties:**
|
||||
- Bounds plain Kolmogorov complexity: K(X) ≤ K^t(X)
|
||||
- Time-bounded so theoretically computable (though intractable in practice)
|
||||
- For any X, K^t(X) ≤ K(X) + log t
|
||||
|
||||
**Failure for randomness:** K^t(prng_output) is small because PRNG has short program + bounded time. So PRNG output is "simple" by Levin complexity, but is "random-looking" by Shannon.
|
||||
|
||||
### D.5 Epiplexity (Formal Definition Attempt)
|
||||
|
||||
> Epi_K(X) = min{K(p) : U(p) outputs X in time ≤ t, K(p) ≤ K}
|
||||
|
||||
Where K is the complexity bound for the program description itself. This is the rough idea — the exact formal definition is in the paper.
|
||||
|
||||
**Comparison to related measures:**
|
||||
- Shannon entropy: H(X) — absolute, observer-independent
|
||||
- Kolmogorov complexity: K(X) — absolute, observer-independent
|
||||
- Levin complexity: K^t(X) — time-bounded, observer-independent
|
||||
- Epiplexity: Epi_K(X) — description-length-bounded, observer-dependent (via K)
|
||||
|
||||
### D.6 Sophistication (Gács-Troutl-Therien)
|
||||
|
||||
**Definition:** Soph(X) = min{K(S) : X ∈ Random(S)}
|
||||
|
||||
Where Random(S) is the distribution of X conditioned on X being "typical" for S.
|
||||
|
||||
**Problem:** The min is over all S, so Soph(X) can be very small or very large. Without computation bounds, complex objects can have low Sophistication.
|
||||
|
||||
**Empirical observation:** Sophistication becomes constant when time-bounded. This is why Wilson/Finzi developed epiplexity as a more practical alternative.
|
||||
|
||||
### D.7 Martin-Löf Randomness (Detailed)
|
||||
|
||||
**Definition:** X is Martin-Löf random iff for all computable (effective) statistical tests T, μ(T(X) rejects) < ε for any ε > 0.
|
||||
|
||||
Equivalently: X passes all effectively constructible statistical tests.
|
||||
|
||||
**Properties:**
|
||||
- Martin-Löf random sequences exist (almost surely for uniform measure)
|
||||
- Computably enumerable tests can be enumerated
|
||||
- Doesn't depend on observer's specific computation
|
||||
- Equivalent to: K(X|n) ≥ n for all n (passes all "compressor" tests)
|
||||
|
||||
**Limitation:** Doesn't capture observer-bounded randomness. A sequence could be Martin-Löf random but still distinguishable from uniform by a sufficiently bounded adversary.
|
||||
|
||||
### D.8 Cryptographic Randomness (Detailed)
|
||||
|
||||
**Definition:** A distribution D on {0,1}^n is (t, ε)-pseudorandom if for all circuits C of size ≤ t, |Pr[C(D) = 1] - Pr[C(U) = 1]| < ε where U is uniform.
|
||||
|
||||
**Key property:** Indistinguishability is RELATIVE to bounded observers. The same distribution can be "random" for some observers and "structured" for others.
|
||||
|
||||
**Connection to epiplexity:** Cryptographic pseudorandom generators ARE structured (short description) but APPEAR random to bounded observers. Epiplexity formalizes this intuition.
|
||||
|
||||
### D.9 Time-Bounded Distinguishability
|
||||
|
||||
The key insight from cryptography and from epiplexity:
|
||||
|
||||
> Time-bounded observer O cannot distinguish X from Y in time t if for all O, |Pr[O(X) = 1] - Pr[O(Y) = 1]| < ε(t)
|
||||
|
||||
This is the practical version of indistinguishability. Different observers (different t bounds) may give different answers.
|
||||
|
||||
### D.10 Generalization Bound Sketch
|
||||
|
||||
For a neural network N trained on dataset D:
|
||||
|
||||
> |Generalization Gap| ≤ O(sqrt(Epi_K(D) / n))
|
||||
|
||||
Where n is training set size and K is the network's complexity.
|
||||
|
||||
**Intuition:** If the training data has low epiplexity (relative to the network's computation), the network can't extract much "weird" structure from it, so it generalizes well.
|
||||
|
||||
This is a strict improvement over classical PAC bounds (which use VC dimension or Rademacher complexity).
|
||||
|
||||
---
|
||||
|
||||
## Appendix E: Connections to Other Frameworks
|
||||
|
||||
### E.1 Predictive Coding
|
||||
|
||||
Predictive coding in neuroscience: the brain maintains a model of the world and updates it based on prediction errors. The "prediction error" is related to the Kullback-Leibler divergence between predicted and actual observations.
|
||||
|
||||
**Connection to epiplexity:** Predictive coding is observer-dependent (the brain has specific computational resources). The "compression" of observations is bounded by the brain's neural architecture.
|
||||
|
||||
### E.2 Minimum Description Length (MDL)
|
||||
|
||||
MDL principle: the best model of data is the one that minimizes the combined length of model + data given model. Formally:
|
||||
|
||||
> L(D, M) = L(M) + L(D | M)
|
||||
|
||||
Connection to Kolmogorov complexity: L(M) ≈ K(M), L(D|M) ≈ K(D|M).
|
||||
|
||||
**Connection to epiplexity:** MDL is computation-unaware. Epiplexity adds the observer's computational constraints to MDL.
|
||||
|
||||
### E.3 Algorithmic Statistics (Vereshchagin-Vitányi)
|
||||
|
||||
Algorithmic statistics: represent an object X via a pair (model, data) such that X is typical for the model. The total code length is bounded.
|
||||
|
||||
Connection to sophistication: Algorithmic statistics refines sophistication by considering both model complexity AND the "regularity" of X within the model.
|
||||
|
||||
**Connection to epiplexity:** Algorithmic statistics provides a framework for "useful structure" but again doesn't account for computation explicitly.
|
||||
|
||||
### E.4 PAC Learning
|
||||
|
||||
Probably Approximately Correct (PAC) learning: a hypothesis class H is PAC-learnable if there's an algorithm that, for any ε, δ, produces a hypothesis h ∈ H with error ≤ ε with probability ≥ 1-δ, using polynomial samples.
|
||||
|
||||
The sample complexity is bounded by VC dimension or similar.
|
||||
|
||||
**Connection to epiplexity:** Epiplexity may provide tighter generalization bounds by accounting for the specific structure of the data.
|
||||
|
||||
### E.5 Algorithmic Information Dynamics
|
||||
|
||||
A more recent framework by Zenil: studies the algorithmic dynamics of computable processes. Treats objects as generated by computable functions and analyzes their evolution under perturbation.
|
||||
|
||||
**Connection to epiplexity:** Algorithmic information dynamics emphasizes the computational nature of information. Epiplexity quantifies this.
|
||||
|
||||
### E.6 Computational Complexity Classes
|
||||
|
||||
P, NP, PSPACE, etc. classify problems by the resources needed to solve them.
|
||||
|
||||
**Connection to epiplexity:** The observer's computational resources (parameterized by K in Epi_K) is itself a complexity class. Epiplexity is "complexity-class-relative information."
|
||||
|
||||
### E.7 PAC-Bayes
|
||||
|
||||
PAC-Bayes: a framework that combines PAC learning with Bayesian methods. Bounds the expected risk of a randomized hypothesis in terms of its KL divergence from a prior.
|
||||
|
||||
**Connection to epiplexity:** PAC-Bayes already accounts for some computational aspects (the choice of prior). Epiplexity provides a more fundamental accounting.
|
||||
|
||||
---
|
||||
|
||||
**End of Report.**
|
||||
|
||||
Lines: ~1,000+ markdown (within target)
|
||||
**"@
|
||||
@@ -0,0 +1,28 @@
|
||||
# Summary: From Entropy to Epiplexity
|
||||
|
||||
**Title:** From Entropy to Epiplexity
|
||||
**Authors/Speakers:** Andrew Wilson and Marc Finzi (with Shikai, Yiding, Pavel, Zico)
|
||||
**YouTube:** https://youtu.be/_U8AwUq_aJQ
|
||||
**Cluster:** A (Math & information-theoretic foundations)
|
||||
**Length:** ~30+ minutes
|
||||
**Type:** Research paper presentation
|
||||
|
||||
## Summary
|
||||
|
||||
Andrew Wilson presents joint work with Marc Finzi on "epiplexity" — a new measure of information that explicitly accounts for the observer's computational resources. The lecture opens with a striking image: "epiplexity" text buried in white noise. Whether you see it depends on your visual computation. Pseudorandom numbers are indistinguishable from random for polynomial-time observers. This is the key insight: randomness is observer-relative.
|
||||
|
||||
The authors present three apparent paradoxes that classical information theory cannot resolve:
|
||||
|
||||
1. **Deterministic processes and information** — Data processing inequality says H(f(X)) ≤ H(X), yet pseudorandom numbers are everywhere and AlphaZero learns sophisticated strategies from deterministic self-play. How can deterministic processes generate information?
|
||||
|
||||
2. **Factorization order** — Symmetry of information says order shouldn't matter, yet LLMs learn far more from left-to-right English than from shuffled English. The order matters.
|
||||
|
||||
3. **Absolute vs. relative information** — Kolmogorov complexity is observer-independent, yet natural images and white noise are visually distinguishable. Existing measures don't separate structure from randomness.
|
||||
|
||||
Classical measures (Shannon entropy, Kolmogorov complexity, Levin complexity, sophistication, Martin-Löf randomness) all fail because they don't ACCOUNT FOR COMPUTATION. Epiplexity fixes this by making the observer's resources an explicit parameter: Epi_K(X) measures information of X relative to an observer with description-complexity bound K.
|
||||
|
||||
The lecture connects to the authors' broader research: using Kolmogorov complexity for neural network generalization bounds, the role of computation in emergence and induction, and connections to Levin search. Pseudorandom numbers have low epiplexity (their generator is short); natural images have low epiplexity for visual computation; LLMs extract epiplexity from left-to-right text.
|
||||
|
||||
The video is foundational for the A-cluster's extension of classical probability/information theory. Wilson/Finzi propose a measure that bridges information theory and computational complexity.
|
||||
|
||||
See [report.md](./report.md) for the full 1,000+ LOC deep-dive with formal definitions, transcript excerpts, frame analysis, mathematical foundations, and connections to other videos in the campaign.
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Quick dedup pass for entropy_epiplexity (frames extracted but not deduped)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[4]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from PIL import Image
|
||||
import imagehash
|
||||
|
||||
|
||||
def main() -> int:
|
||||
frames_dir = ROOT / "conductor" / "tracks" / "video_analysis_entropy_epiplexity_20260621" / "artifacts" / "frames"
|
||||
frame_files = sorted(frames_dir.glob("frame_*.jpg"))
|
||||
print(f"Total frames: {len(frame_files)}")
|
||||
saved_hashes: list[str] = []
|
||||
kept_files: list[str] = []
|
||||
for fp in frame_files:
|
||||
img = Image.open(fp)
|
||||
h = str(imagehash.phash(img))
|
||||
if any(_hamming(h, s) < 5 for s in saved_hashes):
|
||||
fp.unlink()
|
||||
continue
|
||||
saved_hashes.append(h)
|
||||
kept_files.append(fp.name)
|
||||
print(f"Kept: {len(kept_files)}")
|
||||
meta = {
|
||||
"video": "video.mp4",
|
||||
"threshold": 0.05,
|
||||
"total_extracted": len(frame_files),
|
||||
"kept": len(kept_files),
|
||||
"files": kept_files,
|
||||
}
|
||||
(frames_dir / "extraction_meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")
|
||||
return 0
|
||||
|
||||
|
||||
def _hamming(a: str, b: str) -> int:
|
||||
if len(a) != len(b):
|
||||
return max(len(a), len(b))
|
||||
return sum(1 for x, y in zip(a, b) if x != y)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -17,7 +17,7 @@ ROOT = Path(__file__).resolve().parents[4]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from scripts.video_analysis.download_video import download_video
|
||||
from scripts.video_analysis.extract_transcript import _fetch_raw_transcript
|
||||
from scripts.video_analysis.extract_transcript import _fetch_via_ytdlp as _fetch_raw_transcript
|
||||
|
||||
|
||||
def _parse_vtt_segments(vtt_path: Path) -> list[dict]:
|
||||
@@ -48,18 +48,18 @@ def phase1_acquire(slug: str, url: str, artifacts_dir: Path) -> dict:
|
||||
return {"status": "error", "error": f"Could not parse video_id from {url}"}
|
||||
video_id = m.group(1)
|
||||
|
||||
print("Step 1: extract_transcript (try youtube-transcript-api)")
|
||||
print("Step 1: extract_transcript (yt-dlp VTT directly)")
|
||||
transcript_path = artifacts_dir / "transcript.json"
|
||||
last_exc = None
|
||||
for attempt in range(3):
|
||||
try:
|
||||
segments = _fetch_raw_transcript(video_id)
|
||||
segments = _fetch_raw_transcript(video_id, artifacts_dir)
|
||||
data = {
|
||||
"video_id": video_id,
|
||||
"segments": segments,
|
||||
"plain": "\n".join(s["text"] for s in segments),
|
||||
"fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"source": "youtube-transcript-api",
|
||||
"source": "yt-dlp-vtt",
|
||||
}
|
||||
transcript_path.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f" OK: wrote {transcript_path} ({len(segments)} segments)")
|
||||
@@ -72,37 +72,16 @@ def phase1_acquire(slug: str, url: str, artifacts_dir: Path) -> dict:
|
||||
time.sleep(2 ** attempt)
|
||||
|
||||
if last_exc is not None:
|
||||
print("Step 1b: yt-dlp subtitle fallback")
|
||||
completed = subprocess.run(
|
||||
["yt-dlp", "--write-auto-subs", "--sub-langs", "en", "--sub-format", "vtt",
|
||||
"--skip-download", "--output", str(artifacts_dir / video_id), url],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
candidates = list(artifacts_dir.glob(f"{video_id}*.vtt"))
|
||||
if not candidates:
|
||||
print(f" yt-dlp subtitle fetch failed: {completed.stderr[:300]}")
|
||||
print(f" No transcript available. Continuing with download only.")
|
||||
transcript_path.write_text(json.dumps({
|
||||
"video_id": video_id,
|
||||
"segments": [],
|
||||
"plain": "",
|
||||
"fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"source": "unavailable",
|
||||
"error": str(last_exc)[:500],
|
||||
"note": "youtube-transcript-api and yt-dlp VTT both failed. Frame OCR will be the primary signal.",
|
||||
}, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
else:
|
||||
vtt_path = candidates[0]
|
||||
segments = _parse_vtt_segments(vtt_path)
|
||||
data = {
|
||||
"video_id": video_id,
|
||||
"segments": segments,
|
||||
"plain": "\n".join(s["text"] for s in segments),
|
||||
"fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"source": "yt-dlp-vtt",
|
||||
}
|
||||
transcript_path.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
print(f" OK: parsed {len(segments)} segments from {vtt_path.name}")
|
||||
print(f" yt-dlp VTT fetch failed after 3 attempts. No transcript available.")
|
||||
transcript_path.write_text(json.dumps({
|
||||
"video_id": video_id,
|
||||
"segments": [],
|
||||
"plain": "",
|
||||
"fetched_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"source": "unavailable",
|
||||
"error": str(last_exc)[:500],
|
||||
"note": "Frame OCR will be the primary signal for this video.",
|
||||
}, indent=2, ensure_ascii=False), encoding="utf-8")
|
||||
|
||||
print("Step 2: download_video")
|
||||
video_path = artifacts_dir / "video.mp4"
|
||||
|
||||
Reference in New Issue
Block a user