The Problem

Content creators and newsroom operators need a single, repeatable workflow that pulls curated AI‑generated news, injects multimedia, renders a web page and pushes it to a site. Manual ingestion of dozens of feeds is error‑prone and slows publishing.

What This Does

The repo is a daily pipeline that gathers articles, curates them, generates images and a podcast, renders static HTML and deploys the result. The core orchestrator is scripts/run.sh; it pulls raw items via the scout skills, hands them to the editor skill, triggers image generation, writes edition.json, renders newspaper.html from the template/ files, and finally pushes the output to a Git repo.

How It Is Wired

Entry Point – scripts/run.sh The script is executed by scripts/cron_wrapper.sh on a daily cron schedule.

Core Flow –

  1. scripts/run.sh calls the 9 scout skills (skills/scout-…/SKILL.md) one after another, each invoking a Python script under scripts/content/.
  2. The scouts write JSON files to the working directory; scripts/run.sh assembles them into edition.json.
  3. scripts/run.sh invokes the editor skill (skills/editor/SKILL.md) which curates the edition.
  4. Image generation is triggered by the image‑gen skill (skills/image-gen/SKILL.md), which writes image files to assets/.
  5. scripts/render.py reads edition.json and the template/ files to produce output/newspaper.html.
  6. The podcast‑pill skill (skills/podcast-pill/SKILL.md) generates a podcast file using an external TTS service.
  7. Wire‑articles, ticker, and other post‑processing scripts (scripts/…) alter the JSON or inject sections.
  8. Finally, scripts/run.sh commits the rendered artefacts to a remote Git branch for deployment.

What It Touches Outside Itself –

  • Network: HTTP requests to news APIs, YouTube, X/Twitter, and the xAI Grok Imagine/TTS endpoints.
  • Filesystem: reads raw JSON, writes edition.json, images, and the final newspaper.html.
  • Git: commits and pushes the final edition.

File‑by‑File Responsibility –

  • scripts/run.sh: orchestration logic, variable definitions, pipeline control.
  • scripts/core/render.py: deterministic rendering of HTML.
  • skills/*/SKILL.md: LLM‑based decision logic (scouting, editing, injection).
  • scripts/content/*.py: data fetching (e.g., fetch_free_models.py, fetch_trending.py).
  • template/: HTML skeletons and CSS.
  • .github/workflows/validate.yml: CI that runs scripts/validate_repo.py for linting.

The module graph has a single entry point (run.sh) and a linear call chain; no hidden cycles are present.

How To Use It

# Clone the repo verbatim
git clone https://github.com/moses-y/lux-in-tenebris-pipeline

# Install dependencies
pip install -r requirements.txt

# Create a secrets file (see docs/SETUP.md for required env vars)
# .env.example contains placeholders
cp .env.example .env

# Run the pipeline manually
bash scripts/run.sh

The README and docs/SETUP.md lay out the exact environment variables required (PIPELINE_PROVIDER, PIPELINE_MODEL, API keys for news sources, xAI credentials, etc.).

Real‑World Use

A small editorial team can run bash scripts/run.sh nightly on a server, capture the generated output/newspaper.html, and push it to a static hosting service. The pipeline can be forked and the scout skills swapped out to curate a different topic area, preserving the core rendering and deployment logic.

Code Health & Issues

  • Medium (SDLC) – No test files detected; untested code paths exist throughout the repo.
  • Low (Build reproducibility) – requirements.txt lacks a lockfile; builds may vary across environments.
  • Low (CI coverage) – Only linting via .github/workflows/validate.yml; no integration or unit tests.

The Bottom Line

The pipeline cleanly separates LLM‑driven judgment (skills) from deterministic plumbing (scripts), making it easy to swap content sources or add new injection points. Its lack of tests and absence of a dependency lockfile are the main risks. Teams that value rapid, daily content delivery and can tolerate manual validation will benefit most.