The Problem
Standard curl or wget downloads only return flattened HTML. JavaScript-heavy sites render content dynamically, so the actual CSS, JS, and assets a browser loads are invisible to simple HTTP fetchers. Teams doing offline analysis, web archiving, or frontend debugging need the real resource tree a browser would load—not just the initial HTML document.
What This Does
pagesource is a Python CLI that drives a headless Chromium browser via Playwright to capture every network resource a page loads—HTML, CSS, JS, images, fonts—and saves them to disk preserving the original directory structure. The core logic lives in src/pagesource/downloader.py (resource capture and saving), src/pagesource/browser.py (Playwright browser setup), and src/pagesource/cli.py (argument parsing and orchestration). src/pagesource/utils.py handles path sanitization, filename deduplication, and extension inference from Content-Type headers.
The tool strips query strings from filenames, handles duplicate resource names, and optionally includes external/CDN resources under their own host directories. The --wait flag gives JavaScript-heavy SPAs extra time to load before capture.
How To Use It
Setup: Install via pip, then install the Playwright Chromium browser. The pyproject.toml declares Python 3.10+ and Playwright as dependencies.
pip install pagesource playwright install chromium
Configuration: No environment variables or config files required. All behavior is controlled via CLI flags documented in README.md.
Running it: The entry point is the pagesource console command (defined in pyproject.toml), backed by src/pagesource/cli.py.
Basic capture
pagesource https://example.com
Custom output directory, extra wait time, include external resources pagesource https://example.com -o ./output --wait 3 --include-external
Real-World Use
A frontend team investigating a production issue where styles differ between local and deployed environments can capture the actual deployed page's resources:
pagesource https://staging.example.com -o ./staging-capture --include-external
The resulting directory tree shows exactly which CSS/JS files the browser loaded, from which hosts, making it straightforward to compare against local builds or archive a snapshot for regression testing.
Code Health & Issues
Med - No test suite - No test files detected anywhere in the repository. Core logic in downloader.py (path sanitization, filename deduplication, extension inference) is exactly the kind of code that benefits from unit tests. Med - No CI/CD pipeline - No .github/ workflows or CI configuration found. There's no automated gate for linting, testing, or packaging. Low - No dependency lockfile - pyproject.toml declares dependencies but no lockfile is present, so installs aren't fully reproducible across environments. Low - Committed pycache artifacts - .pyc files are present in the repo structure, indicating the .gitignore may not be excluding them properly in all cases.
The codebase is small (5 Python files) and the README is clear and accurate. The lack of tests is the most significant gap; a tool that manipulates filesystem paths and handles network resources would benefit from at least basic coverage.
The Bottom Line
A focused, well-documented CLI tool that solves a real problem for anyone needing the actual resource tree of a webpage. The Playwright dependency makes setup heavier than a pure-HTTP tool, but that's the tradeoff for capturing JavaScript-rendered content. Suitable for web developers, QA engineers, and archivists—though adding tests and CI would make it production-grade.