The Problem

Downloading every file listed in an Internet Archive (archive.org) details page is tedious. The page only exposes an XML index; users must manually parse it, create directories, resume interrupted downloads, and verify hashes.

What This Does

ia-get is a small Rust CLI that accepts a details‑page URL, fetches the associated XML, and downloads every listed object while preserving the archive’s directory layout. Core logic lives in src/main.rs (argument parsing, URL validation, fetch_xml_metadata) and src/downloader.rs (file‑system preparation, hash checking, progress UI). Helper utilities such as create_progress_bar and sanitize_filename are in src/utils.rs.

How It Is Wired

Execution starts at main in src/main.rs (line 130). It validates the input URL (is_url_accessible), resolves the XML URL (get_xml_url), and calls fetch_xml_metadata. fetch_xml_metadata (line 74) parses the XML via parse_xml_files in src/archive_metadata.rs, producing a list of XmlFile structs. The list is handed to download_files in src/downloader.rs.

download_files orchestrates the download pipeline:

  1. setup_signal_handler – prepares graceful shutdown.
  2. For each XmlFile: ensure_parent_directories creates required paths. check_existing_file runs calculate_md5 (which creates a progress bar) to decide whether to skip or resume. download_file_content streams the remote bytes, uses create_progress_bar and format_transfer_rate for UI, and writes to disk. verify_downloaded_file recomputes the MD5 and compares it to the metadata.

The most used symbols are sanitize_filename (called from 18 places) and create_progress_bar (2 places). No circular imports are present; the internal call graph contains 37 edges, all resolvable. The only measured cognitive‑load hot spot is deep nesting in src/downloader.rs (max indentation depth 6).

How To Use It

# Clone the repository
git clone https://github.com/moses-y/ia-get
cd ia-get

# Build the binary
cargo build --release

# Run the tool (replace <identifier> with a real archive.org ID)
./target/release/ia-get https://archive.org/details/<identifier>

No external configuration files or environment variables are required; the program derives all needed URLs from the command‑line argument.

Real‑World Use

A backup service for a digital‑preservation team can schedule a nightly cron job that runs ia-get against a curated list of archive IDs, storing the results in a version‑controlled data lake. Example script fragment:

#!/usr/bin/env bash
ids=("deftributetozzap64" "zzapp_64_issue_001_600dpi")
for id in "${ids[@]}"; do
    ia-get "https://archive.org/details/${id}"
done

The tool’s resume capability avoids re‑downloading large files after network hiccups.

Code Health & Issues

  • High – GitHub Actions are pinned to tags (dtolnay/rust-toolchain@stable, etc.). Tags can move, exposing the CI to supply‑chain risk.
  • Medium – GITHUB_TOKEN permissions are not declared; the workflow inherits full repository rights.
  • Medium – No dependency‑vulnerability scan is run in CI.
  • Medium – Checkout step retains the token (persist-credentials defaults to true), allowing malicious post‑install scripts to push commits.
  • Medium – Deep nesting in src/downloader.rs (max indent 6) makes control flow hard to follow.
  • Low – No job timeout is set; a hung step could block subsequent runs.
  • Medium – No test files are present, meaning core paths are untested.

All findings come from deterministic static analysis; no additional issues were inferred.

The Bottom Line

ia-get delivers a focused, Rust‑based solution for bulk downloading from archive.org with built‑in resume and hash verification. The codebase is small and well‑structured, but it lacks automated tests and has several CI hardening gaps that should be addressed before adopting it in a production pipeline. Suitable for teams that need a quick, command‑line downloader and are comfortable reviewing and tightening the CI configuration.