The Problem

AI agents routinely face two obstacles when consuming web content: raw HTML responses that burn token budgets (4,820+ tokens for complex pages), and 403 errors from aggressive anti-bot defenses. Traditional scrapers require headless browsers (Selenium/Puppeteer), introducing latency, resource overhead, and fragility. webclaw addresses this by extracting clean, structured content at sub-millisecond speeds using Chrome-level TLS fingerprinting without a browser dependency.

What This Does

webclaw is a Rust-first content extraction pipeline spanning 149 files across 10 crates. The core logic lives in crates/webclaw-core (42 extractor functions, called from 20 other files) which defines extract_content, parse_selectors, and build_exclude_set. Content passes through crates/webclaw-fetch/src/client.rs (52 functions, 7 types) which handles HTTP fetching, TLS fingerprinting, and extractor dispatch for 30+ target sites including Amazon, GitHub, and Hugging Face. Downstream, crates/webclaw-llm/src/chain.rs (14 functions) formats output for LLM consumption, calling a model for inference and making outbound network calls. The MCP server at crates/webclaw-mcp/src/server.rs (22 functions) provides the REST/JSON-RPC interface, routing through firefox_or_build and smart_fetch_llm. Entry point main in crates/webclaw-cli/src/main.rs:2300 reaches 267 functions and orchestrates the full pipeline: fetch → extract → format → output.

Internal call graph shows is_empty invoked from 124 places, map from 115, and extract from 67—indicating wide fan-out where changes to extraction logic ripple extensively. Nine files exceed 200 lines (notably main.rs at 2060 lines), and 213 six-line duplicated blocks appear across 52 files, particularly in CLI and MCP server initialization.

How It Is Wired

Execution starts at main in crates/webclaw-cli/src/main.rs:2300, which calls post → main → extraction pipeline. The shortest path to an outbound effect: main -> map [network via state.fetch]. extract calls fetch (26 edges) then map (19 edges). dispatch_by_name and dispatch_by_url fan out to matches, extract, and map with 23+ edges each. extract -> map -> to_llm_text -> make_result forms the core transformation chain. Fourteen functions call a model for inference; seven read/write a database via cloud.rs. No circular dependencies exist among 104 analyzed code files, but deep nesting (max depth 8) and broad exception handling in benchmarks create maintenance risk.

How To Use It

Setup:

  • cargo install --git https://github.com/moses-y/webclaw.git webclaw-cli (from source) or brew install webclaw (Homebrew) or download prebuilt binaries from GitHub Releases
  • Docker available via Dockerfile and Dockerfile.ci

Configuration:

  • Environment variables in env.example; no secrets committed (verified: none found)

Running it:

# CLI extraction
webclaw fetch https://example.com

# MCP server
webclaw mcp serve

Real-World Use

In an AI agent workflow, a Claude Code session calls fetch() and receives a 403 or 142KB HTML blob. Switching to webclaw fetch https://example.com returns clean markdown (≈1,590 tokens, 67% reduction) with preserved links, images, and metadata. The MCP server can be embedded in tool-use pipelines: batch in crates/webclaw-mcp/src/server.rs:329 reaches 147 functions for concurrent scraping. A typical agent loop: fetch URL → extract structured data → feed summarized content to LLM context window, avoiding token overflow and 403 failures.

Code Health & Issues

  • HIGH - Pin third-party GitHub Actions to commit SHAs: .github/workflows uses dtolnay/rust-toolchain@stable, Swatinem/rust-cache@v2, softprops/action-gh-release@v2, docker/setup-qemu-action@v3 (tags can move, risking secret exposure)
  • HIGH - Upgrade pinned dependencies with CVEs: tokio@1 CVE-2021-45710, regex@1 CVE-2022-24713, once_cell@1 CVE-2019-16141 (advisories describe exploitable states of installed versions)
  • HIGH - Open PRs instead of pushing to default branch: .github/workflows/deps.yml uses git push landing directly on the deploying branch
  • MEDIUM - Declare least-privilege GITHUB_TOKEN permissions in CI workflows (2 workflows have no declaration)
  • MEDIUM - Enable Dependabot/Renovate: 9 manifests, no update bot configured
  • MEDIUM - Pin container base image by digest: Dockerfile uses rust:1.93-bookworm, ubuntu:24.04 (mutable tags)
  • MEDIUM - Gate PRs on dependency vulnerability scan: no dependency-review-action in CI
  • MEDIUM - Set persist-credentials: false on checkout: release.yml keeps token in .git/config
  • MEDIUM - Add non-root USER to Docker image: no USER directive found
  • MEDIUM - Test suite insufficient: 1 test file against 100 source files (ratio 0.010)
  • LOW - 30+ measured cognitive load and clarity findings (deep nesting, duplicated blocks, oversized files)

The Bottom Line

webclaw delivers on its promise: fast, local-first web extraction with 67% token reduction and no browser dependency. The Rust implementation is efficient and the modular crate structure (webclaw-core, webclaw-fetch, webclaw-llm, webclaw-mcp) enables composition. However, the codebase shows significant technical debt: oversized files (2,060 lines in main.rs), pervasive duplication (213 blocks), and critical SDLC gaps (pinned Actions tags, unpinned CVEs, missing Dependabot). It's suitable for teams needing extraction performance and willing to maintain the CI/CD pipeline, but the health findings require attention before production deployment at scale.