ArchiveBox

The Problem

ArchiveBox solves the pain of web content disappearing by providing a self‑hosted archive that captures URLs, browser history, bookmarks, Pocket/Pinboard feeds, and more in durable formats (HTML, PDF, WARC, SQLite, etc.). Organisations and individuals use it to preserve evidence, backup social‑media media, and keep research papers accessible long term.

What This Does

ArchiveBox is a Django‑backed Python application that accepts input via CLI, browser extension, or API, then persists the crawled content to the filesystem and a SQLite database. Core logic lives in archivebox/core/models.py (196 functions, 13 types) which defines the data model (get_group_by_cols, slug, api_url, to_json) and is called from 120 other files. Input handling starts at archivebox/cli/__init__.py:191 (main) and archivebox/core/management/commands/archivebox.py:15 (handle), both of which delegate to run_archivebox_cmd (called 281 times) and dispatch (called 125 times). Crawling and media extraction are orchestrated by archivebox/crawls/models.py (68 functions, 5 classes) and archivebox/machine/models.py (95 functions, 17 classes), each reading/writing the database and filesystem. Dockerisation is provided by Dockerfile and docker-compose.yml, while configuration is stored in archivebox/config/common.py (62 functions, 8 classes) and read via get_config (123 callers). The project ships with a browser‑extension exporter, a REST API (archivebox/api/v1_core.py), and a web interface served through the Django ORM.

How It Is Wired

Execution flows from the entry points through a small set of hub modules:

Key paths to external side‑effects:

• dispatch → _latest_snapshot_for_url → find_snapshots_for_url – reads Snapshot.objects.filter (database).
• handle → current – reads cls.objects.get (database).
• cli → setup_django – spawns a subprocess (subprocess.run).
• init → set – reads Binary.objects.filter(id=binary_id).afirst (database).
• main → run_runner – filters Crawl.objects.filter(id=crawl_id, status__in=Crawl.RUNNABLE_…) (database) then runs external commands via archivebox/services/runner.py.

The import graph contains 52 modules in circular dependencies (e.g., archivebox/core/models, archivebox/crawls/models, archivebox/machine/models, archivebox/config/common, archivebox/config/__init__, archivebox/services/runner). These cycles increase blast radius: a change to archivebox/core/models.py ripples through 97 importers, and archivebox/tests/conftest.py is imported by 81 other modules.

How To Use It

Setup

• pip: pip install archivebox (pyproject.toml indicates pip/uv).
• Docker: build from Dockerfile or run the published image archivebox/archivebox. docker-compose.yml defines the full stack (Postgres, Nginx, the ArchiveBox container).

Configuration

• Environment variables and settings live in archivebox/config/common.py and are read by get_config. No secret is committed; the repo states “committed secrets: none found”.

Running it

• CLI: archivebox add https://example.com (invokes main → run_archivebox_cmd).
• Python API: from archivebox.archive import add_url; add_url("https://example.com").
• Docker quickstart (from README): curl -fsSL 'https://get.archivebox.io' | bash or docker run -p 8000:8000 archivebox/archivebox.

Stopping / migrating

• Management command: archivebox manage migrate (uses Django migrations under archivebox/core/migrations/).

Real‑World Use

A research team needs to preserve a set of open‑access papers and their referenced media. They run archivebox add https://arxiv.org/abs/2305.01234; the CLI enqueues the URL, archivebox/core/models.py creates a Snapshot record, archivebox/crawls/models.py launches a headless browser, extracts HTML, PDF, and embedded images, and stores them under data/snapshots/<id>/. The web UI (archivebox/core/views.py) then serves the archived copy, while the REST endpoint archivebox/api/v1_core.py allows an internal CI job to query the archive status. Because the data lives on disk and in SQLite, the team can migrate the archive to a new host by copying the data/ directory and re‑registering the database.

Code Health & Issues

Measured analysis (static, deterministic):

• 232 total findings: 86 high, 145 medium, 1 low.
• 329 internal modules, 990 import edges, 52 modules in circular dependencies.
• Hub modules: archivebox/core/models.py (97 dependents), archivebox/tests/conftest.py (81 dependents), archivebox/crawls/models.py (71 dependents).
• Deep nesting (indentation depth 8) in archivebox/core/models.py, archivebox/crawls/models.py, archivebox/machine/models.py.
• Oversized files (> 3 k lines) same as above.
• Broad exception handling (except Exception) in archivebox/core/models.py, archivebox/crawls/models.py, archivebox/config/common.py.
• Import‑cycle members in the same three files plus archivebox/config/common.py.
• File opened without context manager in archivebox/tests/conftest.py, archivebox/services/runner.py, archivebox/workers/supervisord_util.py.

CODE HEALTH AUDIT (6 findings, each with a prescribed fix):

• [HIGH] Pin third‑party GitHub Actions to a commit SHA – .github/workflows uses @v3, @v6 etc.; replace with 40‑char SHA and let Dependabot bump them.
• [HIGH] Commit a lockfile beside the manifest – etc/package.json has no lockfile; run the package manager once and commit the generated file.
• [MEDIUM] Pin the container base image by digest – Dockerfile references archivebox/sonic:1.4.9, ubuntu:24.04; pin to image:tag@sha256:<digest> and enable Dependabot Docker scanning.
• [MEDIUM] Gate pull requests on a dependency vulnerability scan – no dependency‑review action in CI; add dependency-review-action on pull_request or osv-scanner on push.
• [MEDIUM] Add a non‑root USER to the image – Dockerfile ends with CMD/ENTRYPOINT without a USER directive; create an unprivileged user, chown needed paths, and end the Dockerfile with USER.
• [LOW] Set timeout-minutes on workflow jobs – four jobs in .github/workflows/deploy-publicsite.yml have no timeout; add a realistic bound to each job.

The Bottom Line

ArchiveBox is a capable, well‑documented self‑hosted archiving platform that captures web content in durable formats and offers CLI, API, and web UI entry points. Its strengths lie in extensive format support, Docker support, and an active community. Maintainability suffers from deep import cycles, oversized core modules, and several hygiene gaps (missing lockfile, mutable base images, unpinned GitHub Action SHAs). Teams that need a flexible, open‑source way to preserve web evidence will find it valuable, but should budget time to refactor the hub modules, add a lockfile, and lock down container base images before deploying to production.