Katana Technical Briefing

The Problem

Web crawling at scale breaks down when targets rely on JavaScript rendering, CAPTCHAs, and complex form flows. Traditional HTTP-only crawlers miss dynamic content, while headless browsers are slow and hard to configure. Katana addresses this by unifying both approaches in one framework.

What This Does

Katana is a Go-based crawling and spidering framework with three execution engines: pkg/engine/standard/ for HTTP-only crawling, pkg/engine/headless/ for browser automation, and pkg/engine/hybrid/ that combines both. The headless engine includes CAPTCHA solving (pkg/engine/headless/captcha/) with support for hCaptcha, reCAPTCHA, and Turnstile, plus automatic form filling (pkg/engine/headless/crawler/formfill.go).

The framework provides output formatting (pkg/output/), scope control (pkg/utils/scope/), and filtering (pkg/utils/filters/). It supports JSON, template, and screen output formats. The cmd/katana/main.go entry point exposes a CLI with flags for input, configuration, and output control.

How It Is Wired

Execution starts at cmd/katana/main.go, which delegates to internal/runner/runner.go. The runner parses options via internal/runner/options.go, then selects an engine through pkg/engine/engine.go. The standard engine (pkg/engine/standard/crawl.go) performs HTTP requests, while the headless engine (pkg/engine/headless/headless.go) drives a browser via pkg/engine/headless/browser/browser.go.

The headless crawler (pkg/engine/headless/crawler/crawler.go) handles page navigation, form filling, and state management (state.go). Captcha detection flows through pkg/engine/headless/captcha/identify.go into solver implementations under pkg/engine/headless/captcha/capsolver/. Parsed results route through pkg/engine/parser/parser.go into pkg/output/output.go for formatting and writing.

The widest blast radius is pkg/engine/parser/parser.go at 635 lines—changes there affect all engines. The import graph shows 6 internal modules with 0 edges, meaning the codebase is functionally modular with no circular dependencies.

How To Use It

Setup:

CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest

Docker:

docker pull projectdiscovery/katana:latest
docker run projectdiscovery/katana:latest -u https://tesla.com

Basic crawl:

katana -u https://example.com

Headless mode with CAPTCHA solving:

katana -u https://example.com -headless -system-chrome

Configuration is flag-based; no config file is required. The Makefile provides build targets, and go.mod declares 120 dependencies.

Real-World Use

For security testing, Katana fits into a pipeline where you need to discover endpoints from a JavaScript-heavy SPA:

katana -u https://app.example.com -headless -jc -d 3 -o endpoints.txt

This crawls three levels deep, parses JavaScript (-jc), and outputs discovered URLs to a file for subsequent testing with tools like nuclei or ffuf.

Code Health & Issues

Static analysis found 25 issues (4 high, 20 medium, 1 low). Key findings:

  • High - Deep nesting - pkg/engine/headless/crawler/formfill.go, headless.go, parser_test.go hit indentation depth 8; control flow is hard to follow. Flatten with early returns.
  • High - Duplicated code - 26 repeated 6-line blocks across 16 files including pkg/engine/common/base.go, hybrid/hybrid.go, browser/browser.go. Extract shared helpers.
  • Medium - Empty catch blocks - pkg/engine/headless/captcha/js/inject-recaptcha.js and page-init.js silently discard errors.
  • Medium - Oversized file - pkg/engine/parser/parser.go at 635 lines.
  • Medium - High branching - internal/runner/options.go has 58 branch points over 161 lines.
  • Low - TODO markers - 3 unresolved in pkg/engine/headless/crawler/crawler.go.

The Bottom Line

Katana is a capable crawler with solid architecture—no circular dependencies, good test coverage (51 test files), and CI. The main maintenance burden is the duplicated logic and deep nesting in the headless engine. Teams needing JavaScript-rendered crawling with CAPTCHA bypass should evaluate it; teams with simple static-site needs may find the standard engine sufficient but heavier than alternatives.