The Problem

Organizations need a repeatable way to collect, correlate, and visualise open‑source intelligence (OSINT) about assets such as domains, IPs, emails or crypto addresses. Doing this manually is error‑prone and scales poorly, especially when dozens of public data sources must be queried and the results merged into a single view.

What This Does

spiderfoot/ ships a Python 3 Flask application that can be run as a web UI or a CLI scanner. The core entry point is spiderfoot/spiderfoot.py, which parses command‑line arguments, instantiates the SpiderFoot engine, loads every module found under modules/, and drives the scan loop. Over 200 data‑source modules (e.g. modules/sfp_shodan.py, modules/sfp_github.py) each implement a handleEvent method that queries a specific API or scrapes a web page.

Correlation rules live as YAML files in correlations/ (e.g. correlations/email_in_multiple_breaches.yaml). After modules emit events, the engine feeds them to the correlation engine, which applies these rules and produces higher‑level alerts that are shown in the UI or exported as CSV/JSON/GEXF.

How It Is Wired

Entry point

File: spiderfoot/spiderfoot.py Function: main() – uses argparse to decide between --cli (CLI mode) or Flask app.run() (web UI).

Engine bootstrap

main() creates a SpiderFoot object (defined in the same file). The constructor:

  1. Calls load_modules() – iterates over modules/, imports each sfp_*.py, and registers the class in self.modules.
  2. Calls load_correlations() – reads every *.yaml under correlations/ into self.correlation_rules.

Scan flow (CLI)

  1. run_scan(target) → creates an empty EventStore.
  2. For each loaded module, module.start(target, self.event_store) is invoked.
  3. Inside a module (e.g. modules/sfp_shodan.py), handleEvent(event) builds an HTTP request, parses JSON, and calls self.sfdb.add_event(new_event).
  4. After all modules finish, apply_correlations() walks self.correlation_rules; each rule’s Jinja‑style condition is evaluated against the EventStore. Matching rules generate Alert objects stored back in the DB.

Persistence & output

File: modules/sfp__stor_db.py – implements a SQLite backend (self.conn = sqlite3.connect('sf.db')). All events and alerts are written here, making them available to the Flask UI (spiderfoot/templates/) and to export commands (--output csv).

External effects

Network: Every module performs outbound HTTP(S) calls to third‑party APIs; the only shared network entry point is the requests library used in each sfp_*.py. Filesystem: SQLite DB file (sf.db) and optional log files under the working directory; Docker images mount /data for persistence. Process: No background workers; the scan runs synchronously in the invoking process.

Hotspots

spiderfoot/spiderfoot.py – single entry point; any change to argument handling or engine init ripples through the whole system. modules/sfp__stor_db.py – central persistence; a bug here can corrupt all scan data. Correlation loader – YAML files are interpreted at runtime; malformed rules cause a full scan abort.

No cyclic imports were found; the module graph is a shallow star (engine → many leaf modules). This layout simplifies adding or removing a data source but makes the engine a single point of failure.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/spiderfoot
cd spiderfoot

# Install Python dependencies
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# Optional: build the Docker image (uses Dockerfile)
docker build -t spiderfoot .

# Run the web UI (default on port 5000)
python3 spiderfoot/spiderfoot.py --web

# Or run a one‑off CLI scan against example.com
python3 spiderfoot/spiderfoot.py -t example.com -l   # list modules
python3 spiderfoot/spiderfoot.py -t example.com -s   # start scan

Configuration files for API keys are expected in config.cfg (created on first start) or as environment variables; the README points to the “API Keys” section for each module.

Real‑World Use

A security operations team can schedule a nightly Docker container:

# docker‑compose.yml excerpt
services:
  spiderfoot:
    image: spiderfoot
    command: ["python3", "spiderfoot/spiderfoot.py", "-t", "mycorp.com", "-s", "--output", "json"]
    volumes:
      - ./data:/data   # persists sf.db

The resulting JSON feed is ingested by a SIEM to trigger alerts when new breached credentials appear for corporate domains.

Code Health & Issues

  • Low – missing lockfile – dependencies are listed only in requirements.txt; reproducible builds rely on external index state.
  • Low – no pyproject.toml or setup.cfg entry points – the CLI entry point is a script rather than an installable console command, which complicates packaging.
  • Low – large test suite (461 files) but no coverage badge in this fork – CI workflow (.github/workflows/tests.yaml) runs tests, but the fork does not publish coverage results, making quality gating less visible.

No critical security findings are evident from the repository layout; however, the reliance on many external API keys means secret management must be handled outside the codebase.

The Bottom Line

SpiderFoot provides a mature, extensible OSINT engine with a clear separation between data‑source modules, a correlation engine, and a Flask UI. The codebase is straightforward to extend but hinges on a single engine file and a SQLite store, which may limit scalability. It is well‑suited for security teams that need a customizable, on‑premises OSINT scanner and are comfortable managing API secrets and Docker deployments.