The Problem

Many teams need to capture the full set of assets a page loads—CSS, JavaScript, fonts, images—so they can audit, test, or archive the page offline. Existing CLI tools either dump a single flattened HTML file or require custom browser scripts that are fragile and hard to maintain. There was no lightweight, reusable Python utility that preserves the original directory structure while handling query‑string stripping, extension inference, and duplicate‑filename resolution.

What This Does

pagesource is a Python CLI (entry point src/pagesource/cli.py:30) that drives a headless browser via Playwright to navigate a URL, collect every network resource, and write them to disk keeping the URL‑derived path layout.

  • capture (cli.py) is the top‑level function; it reaches 14 other internal functions before the process ends.
  • Resources flow through downloader.py (save_resource, save_resources, ResourceSaver) which writes files via local_path.parent.mkdir and infers extensions with utils.py (infer_extension, is_same_origin, sanitize_path_component).
  • The browser layer (browser.py) orchestrates page load, waits (--wait), and dispatches events to the downloader. It contains a bare except Exception that swallows errors indiscriminately (see Health section).
  • utils.py provides pure‑Python helpers: parse_url, _split_extension, infer_extension, is_same_origin, and sanitize_path_component.

The tool can output to a custom directory (-o) and optionally keep external CDN assets (--include-external) in their own host folders.

How It Is Wired

Execution starts at capture in src/pagesource/cli.py:30. The call graph (16 resolved edges) shows the shortest path to the filesystem:

capture → save_resources → save_resource → local_path.parent.mkdir

Key edges (caller → callee, frequency):

  • capture_page_resources → _status (4×)
  • url_to_local_path → sanitize_path_component (2×)
  • save_resource → is_same_origin, url_to_local_path, infer_extension, _deduplicate_path
  • save_resources → ResourceSaver → save_resource

No edges leave the repository besides the filesystem write and the Playwright API calls that are external to the code base. The module dependency graph has src/pagesource/utils as a leaf (3 importers, 0 importers of others) and src/pagesource/browser as a modest hub (2 importers, 1 importer). Changing the downloader or utils ripple through all callers, but there are no circular dependencies.

How To Use It

Setup

# Install the package
pip install pagesource

# Install the Playwright browser (required)
playwright install chromium

Configuration No external config files or environment variables are required beyond the URL. The only prerequisite is Python 3.10+ and the Playwright Chromium binary.

Running it

# Basic capture
pagesource https://example.com

# Custom output and wait for SPA rendering
pagesource https://example.com -o ./my-output --wait 5

# Include third‑party CDN assets
pagesource https://example.com --include-external

All output lands under ./pagesource_output/<host>/ preserving css/, js/, images/, etc. Query strings are stripped from filenames; duplicate names are disambiguated automatically.

Real‑World Use

A QA team wants to archive a single‑page app for later regression testing. They run:

pagesource https://app.example.com -o ./archive --wait 3 --include-external

The resulting ./archive/ tree contains the app’s bundled JS, CSS modules, and external font files, which can be version‑controlled or fed into an automated accessibility scanner. Because the directory structure mirrors the live site, downstream scripts can locate assets by path without parsing HTML.

Code Health & Issues

  • [HIGH] Commit a lockfile beside the manifest – pyproject.toml declares dependencies without a lockfile, so installed versions can diverge between developer machines and CI/CD. Fix: run the package manager once and commit the generated lockfile.
  • [MEDIUM] Enable Dependabot or Renovate – No update bot is configured; security advisories remain unpatched until a manual audit. Fix: add .github/dependabot.yml for Python ecosystems and GitHub Actions.
  • [MEDIUM/resilience] Broad exception handling – src/pagesource/browser.py uses bare except Exception that swallows errors indiscriminately. Fix: catch specific exceptions (e.g., playwright._impl._errors.PageError, OSError) and log or re‑raise the rest.

No test files or CI/CD pipeline are present, and the repository has no Dockerfile or committed secrets.

The Bottom Line

pagesource fills a niche: a no‑nonsense CLI that extracts a page’s actual assets with minimal configuration. The codebase is small (5 Python files, ~150 lines total) and the call graph is shallow, making it easy to inspect or modify. The main production concerns are the missing lockfile and the unhandled exception in the browser layer; addressing those will raise confidence for any team integrating it into a pipeline. It is well‑suited for one‑off captures, documentation archives, or lightweight asset‑audit workflows.