03 · Content Pipeline
← Docs home · Task: turn a raw textbook section into a published, enhanced HTML page.
Source of truth:
.claude/skills/book-pipeline/SKILL.md(the live spec, ~1000 lines) anddocs/PIPELINE_GUIDE.md(exhaustive internals: chunking, state/resume, performance). This page is the map; those are the territory.
How to run it
The pipeline runs in-session. You tell Claude:
“run book pipeline for 8.4 of Introduction to Stats”
Claude binds PROJECT="Introduction to Stats", SECTION="8.4", CHAPTER=8,
resolves the project’s directory layout, and walks the 15 steps in order —
skipping any step whose output is already fresh (output exists and is newer
than every input).
Always preflight first
Before any pipeline command, the book-pipeline-preflight skill (auto-fires on
“run the pipeline” / “publish the book”) validates every project manifest and
confirms the source files exist:
python scripts/workflows/preflight.py --project "Introduction to Stats" --section 8.4
Exit 0 = green · 1 = warnings · 2 = errors (halt and fix before running).
It checks pipeline_config.yaml, figure_animations.yaml, manim_candidates.yaml,
section_videos.yaml, and source completeness.
The 15 steps (ALL_STEPS)
scrape → merge → extract → normalize-tags → remaster → number → solutions → philosophy-reorg → youtube → math → html → fixup → publish → figure-animator → slides
| # | Step | Kind | Helper / what happens |
|---|---|---|---|
| 1 | scrape | Bash | scrape_concat.py — merge multiple source pages into one MD. |
| 2 | merge | Bash | 3 phases: fingerprint_dedup.py → placement_map.py → merge_from_placement.py (reconcile primary + supplemental sources). |
| 3 | extract | Bash | extract_sections.py — split merged MD into per-section source files. |
| 4 | normalize-tags | Bash | normalize_source_tags.py — convert > **Example X.Y** blockquote tags into ### Example H3s. Idempotent. |
| 5 | remaster | AI | Rewrite source MD to textbook density/format. Prompt: prompts/remaster-chapter.md. Lint: lint_remaster.py (threshold 75/100). Chunk if >800 lines. |
| 6 | number | Bash | number.py — auto-number headings, Examples, TINs, Definitions to Chapter.Section.N. Idempotent. |
| 7 | solutions | AI | Write one <details> worked solution per Problem-Set problem; splice into a copy of the numbered MD via splice_solutions.py. |
| 7.5 | philosophy-reorg | Bash | python -m scripts.workflows.philosophy_reorg — enforce Def → TIN → Example order within each H2. Run on numbered MD and _Solutions.md. |
| 8 | youtube | AI + Bash | Extract topics → write *_video_queries.json; then youtube_lookup.py --scrape fills a top URL per query (powers the Videos menu). |
| 9 | math | AI + Bash | math_verify.py pre-checks (delimiter balance, element counts) + AI review → *_MathVerify.md. |
| 10 | html | AI + Bash | AI writes the body fragment; html_gen.py --body-file wraps it in the skeleton (nav, settings, theme). |
| 11 | fixup | Bash + AI | fixup_phase_a.py (18 deterministic HTML cleanups) + Phase B AI fixes (prompts/fix-html-block.md). |
| DISABLED | Assembled chapter pages retired 2026-06-06 (problem-numbering divergence). Do not run stitch_chapter_html.py. |
||
| 13 | publish | Bash | _verify_html → .verify_ok stamp → publish.py copies to docs/. See 08. |
| 14 | figure-animator | Bash + AI | Auto-runs with publish.py --auto-render when the section has figure_animations.yaml entries → renders MP4/PNG pairs + swaps <img>. See 04. |
| 15 | slides | Bash | publish.py --slides builds a Slidev deck. See 06. |
Step 17 — rashio-step (post-publish, optional)
Not in ALL_STEPS. A post-publish enhancement that runs only when a section
has TI-calculator notes or tasks matching the raSHio tool catalog: (A) replace TI
notes with raSHio equivalents, (B) record a dual-theme Playwright walkthrough
video, © add rashio-note callouts (cap 3/section). Invoke by hand or via the
rashio-step skill. Deploy raSHio before the page ships — the notes link to
the live tool site.
Single-source path (no supplemental book)
When a project has exactly one source (pure OpenStax / OpenIntro / scraped), steps 1–3 don’t apply. Use the bridge:
python3 scripts/workflows/single_source_extract.py \
--project "<project_name>" --source <source_name> --chapter N
It reads per-page MD (source_files/<source>/3-1.md, …), writes the canonical
Chapter_N_Source/ layout, and auto-downloads each figure’s CDN image to
html/figures/. Then proceed straight to step 4 (normalize-tags).
OpenStax sources are extracted by
source_files/openstax/to_markdown.py, which cleans MathML→LaTeX artifacts (glued operators, piecewise envelopes, escaped percent) at extract time so remaster doesn’t have to. If a re-extract drops figures or examples, see theopenstax-audit-healprocedure in 09.
Outputs (where things land)
| Artifact | Path |
|---|---|
| Canonical source | projects/<Book>/remastered/Chapter_N_Source/<sec>_*.md |
| Remastered | projects/<Book>/remastered/Chapter_N_Remastered/<sec>_*.md |
| Numbered + solutions | projects/<Book>/remastered/Chapter_N_Numbered/<sec>_*.md (+ _Solutions.md) |
| Assembled HTML | projects/<Book>/html/<sec>_*.html |
| Published page | docs/<book-slug>/chapter-N-*/<sec>_*.html |
Key principles (carry these in your head)
- Smaller jobs = better AI output. Large sections are chunked (remaster
800 lines, HTML, solutions) so each AI call stays focused. The chunk map is in
PIPELINE_GUIDE.md. - Idempotent + resumable. Re-running is safe; fresh steps skip. State lives in
_pipeline_state.json(and remaster checkpoints). Resume after a crash just by re-invoking. - Deterministic where possible, AI only where needed. Numbering, splicing, HTML assembly, publishing are scripts; remaster/solutions/math/html-body are AI.
- Lint, feed back, retry once. Each AI step runs its lint; on failure it feeds findings back and regenerates once, then logs and continues.
Common pipeline issues
See 10 · Troubleshooting for: subagent
returned unexpected output, remaster freeze, missing _pipeline_state.json,
math-check false positives, and the orphan-_Solutions.md regression.