The Problem

Large‑scale web crawling demands a tool that can ingest billions of URLs, respect politeness policies, and feed results into downstream indexes. Existing crawlers are either monolithic (hard to extend) or lack the batch‑processing model needed for Hadoop‑based pipelines. Teams building custom search pipelines therefore need a proven, extensible framework that can be run both locally and in a distributed environment.

What This Does

Apache Nutch is a modular, Hadoop‑compatible crawler. The source lives under src/java/ (≈ 600 Java files) and is driven by the shell wrappers in src/bin/ – nutch and crawl.

  • src/bin/nutch builds the Java classpath from conf/nutch-default.xml and launches the selected command (inject, generate, fetch, …).
  • Core crawl logic lives in src/java/org/apache/nutch/crawl/ (Injector, Generator, FetcherThread, CrawlDbReader/Writer).
  • Indexing is handled by src/java/org/apache/nutch/indexer/ (IndexerMapReduce, IndexWriter).

Configuration files in conf/ (e.g., nutch-site.xml, log4j2.xml) control URL filters, fetch schedules, and plugin loading. A Docker build (docker/Dockerfile) packages the Java runtime with the source, enabling reproducible runs.

How It Is Wired

Execution starts with the entry point src/bin/nutch. The script parses the first argument to select a Java class (e.g., org.apache.nutch.crawl.Injector).

  1. Injector (src/java/org/apache/nutch/crawl/Injector.java) reads seed URLs from a local file and writes initial CrawlDatum records to the CrawlDB.
  2. Generator (src/java/org/apache/nutch/crawl/Generator.java) scans the CrawlDB, applies the fetch schedule (AbstractFetchSchedule, AdaptiveFetchSchedule), and emits a segment of URLs for fetching.
  3. Fetcher (src/java/org/apache/nutch/fetcher/Fetcher.java) spawns FetcherThread workers (large file, 1153 LOC) that open HTTP connections, respect conf/httpclient-auth.xml, and write fetched content to the segment store.
  4. Parse plugins (e.g., src/plugin/parse-js/) consume the raw content; the most connected module is src/plugin/parse-js/sample/parse_pure_js_test (no imports).
  5. Indexer (src/java/org/apache/nutch/indexer/IndexerMapReduce.java) reads parsed documents and writes to the configured index writer (Elasticsearch, Solr, CloudSearch via plugins).

All stages read/write Hadoop SequenceFiles under the crawl/ directory hierarchy, so the blast radius of a change to FetcherThread or Generator propagates to every crawl segment. No circular import dependencies were detected, keeping the internal module graph simple (1 module, 0 edges).

How To Use It

# Clone the repo
git clone https://github.com/moses-y/nutch
cd nutch

# Build with Ant (Apache Ant is the project’s build tool)
ant clean runtime

# Create a working directory and seed file
mkdir -p crawldb urls
echo "https://example.com" > urls/seed.txt

# Inject seeds
src/bin/nutch inject crawldb urls/seed.txt

# Generate fetch list (default 100 URLs)
src/bin/nutch generate crawldb segments

# Fetch pages
src/bin/nutch fetch segments

# Parse and index (requires an index writer plugin, e.g., solr)
src/bin/nutch index crawldb segments

Configuration tweaks belong in conf/nutch-site.xml (e.g., plugin.folders, http.agent.name). The Docker image can be built with docker build -t nutch . and run the same CLI commands inside the container.

Real‑World Use

A news aggregator can schedule a nightly crawl by invoking the above pipeline in a CI job, storing the generated NutchDocument objects in an Elasticsearch index, then querying that index for fresh articles. The modular plugin system lets the team add a custom HtmlParseFilter to extract article metadata without touching the core crawl code.

Code Health & Issues

Measured findings (static analysis):

  • HIGH – Deep nesting in IndexerMapReduce.java, ParseOutputFormat.java, TableUtil.java (max depth 9).
  • HIGH – Duplicated code across shell scripts and Java schedule classes (2674 × 6‑line blocks).
  • MEDIUM – Oversized files (CrawlDbReader.java, Generator.java, FetcherThread.java > 1100 LOC).
  • MEDIUM – High branching density in 33 files (e.g., UpdateHostDbReducer.java).

Health audit (security/CI):

  • CRITICAL – Untrusted PR code checked out in privileged workflow (.github/workflows/sonarcloud.yml).
  • CRITICAL – Secrets (SONAR_TOKEN) exposed to fork‑triggered runs.
  • HIGH – GitHub Actions not pinned to commit SHAs.
  • HIGH – Steps discard exit codes, masking failures.
  • HIGH – No test execution in CI despite 176 test files.
  • HIGH – continue-on-error on correctness steps in master-build.yml.
  • MEDIUM – Base image not pinned by digest (docker/Dockerfile).
  • MEDIUM – Checkout persists credentials; should set persist-credentials: false.
  • MEDIUM – Container runs as root; add non‑root USER.
  • LOW – No job timeout limits (junit-report.yml).

Other hygiene: tests present, CI configured, Dockerfile exists, license files included, no committed secrets detected.

The Bottom Line

Nutch provides a battle‑tested, plugin‑driven crawler that integrates tightly with Hadoop and can be containerised. The codebase is large and contains several high‑complexity hotspots and security mis‑configurations that should be addressed before production use. Teams comfortable with Java, Ant, and Hadoop will find it extensible; newcomers must first resolve the CI/security issues and consider refactoring the most tangled modules.