Krawl - Moses Yebei

Abstract

Krawl is a customizable, lightweight, cloud-native web deception server and anti-crawler that creates fake web applications with low-hanging vulnerabilities using realistic, randomly generated decoy data and AI-generated HTML templates.

The Problem

Web crawlers, automated scanners, and opportunistic attackers waste resources probing exposed services. Distinguishing malicious traffic from legitimate crawlers is noisy and reactive. Krawl solves this by giving attackers a fake attack surface—decoy apps, fake credentials, and honeypot paths—so you can detect, track, and ban malicious IPs without risking real infrastructure.

What This Does

Krawl is a Python-based deception server. It serves thousands of realistic decoy pages (.env.html, .aws__credentials.html, fake admin panels) from src/templates/deception/, plus spider-trap pages that generate infinite random links. It logs every interaction, scores IP reputation, and can auto-ban via firewall integrations.

The project includes a real-time dashboard (src/routes/dashboard.py), an API (src/routes/api.py), background tasks (src/tasks/), and optional AI-generated deception pages (src/generative_ai.py). Deployment is flexible: Docker Compose, Kubernetes via Helm (helm/), or bare Python with Uvicorn.

How It Is Wired

Execution starts at src/app.py, which initializes the FastAPI app and mounts routes. Middleware in src/middleware/deception.py intercepts requests, checks the banlist, and serves decoy content. The src/routes/honeypot.py handles the actual deception responses. Every request touches src/database/access_logs.py to record the hit.

The import graph shows a mostly flat structure: 55 internal modules with only 2 import edges, and no circular dependencies. The only hub is src/templates/template_loader.py (2 modules import it), which loads HTML templates. This means the codebase is highly decoupled—each module is self-contained, but the trade-off is that the src/database/ modules (analytics, ip_stats) are oversized (790+ lines) and hard to change without wide ripple effects.

External effects: SQLite/PostgreSQL/MariaDB writes via src/database/core.py, firewall commands via src/firewall/iptables.py and src/firewall/nftables.py, and outbound LLM calls in src/generative_ai.py. The scripts/migrate_sqlite_to_*.py files handle database migrations.

How To Use It

Setup (from README):

git clone https://github.com/moses-y/Krawl
cd Krawl
docker compose up -d

For Kubernetes: helm install krawl ./helm (see helm/values.yaml).

Configuration: Edit config.yaml for wordlists, paths, and deception settings. Environment variables override these (documented in README). The .env.example.html is a decoy, not a real config.

Running: docker compose up is the primary path. Bare Python: uvicorn src.app:app from the repo root, but you'll need Redis and a database configured first.

Real-World Use

Deploy Krawl behind a reverse proxy alongside production services (see docs/reverse-proxy.md). Route all unknown paths to Krawl—legitimate crawlers hit decoys, get logged, and can be auto-banned via the banlist.txt integration. An attacker probing /.env gets a fake file with realistic credentials, triggering a canary token alert, and their IP lands on the banlist within minutes.

Code Health & Issues

Static analysis found 37 findings (6 high, 31 medium). Key issues:

High - Duplicated code: 177 repeated 6-line blocks across 16 files, notably scripts/migrate_sqlite_to_mariadb.py and scripts/migrate_sqlite_to_postgres.py. Extract shared helpers.
High - Deep nesting: 22 instances, max depth 8 in src/database/core.py and src/generative_ai.py. Flatten with guard clauses.
Medium - Broad exception handling: 7 instances of bare except in src/database/ip_stats.py, src/metrics_counters.py. Catch specific exceptions.
Medium - Oversized files: src/database/analytics.py (790 lines) and src/database/ip_stats.py. Split by responsibility.

Security audit findings: unpinned GitHub Actions (use commit SHAs), no lockfile (pyproject.toml), CI never runs tests (8 test files, no test command), continue-on-error on a correctness step, no non-root user in Dockerfile, and 1 test file covering 60 source files (ratio 0.017). Committed secrets are decoys—the audit confirms they're template files, not real credentials.

The Bottom Line

Krawl is a functional, well-documented deception tool with solid deployment options. The codebase is decoupled and easy to navigate, but the lack of a lockfile, untested CI, and oversized database modules are real risks if you plan to modify it. Use it if you want a self-hosted honeypot with minimal effort; budget time for CI fixes and test coverage before trusting it in production.

What the analyser found

Deployment readiness

5/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

1,512 files

HTML1335
Python56
YAML33
Markdown21
JavaScript9
Shell5

ReactNext.jsVueDjangoFlaskExpressSpringLaravelDockerKubernetesTerraform

Module dependencies

The 10 most depended-upon modules of 55, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_rc_templates_template_loader["template_loader"] n1_src_templates___init__["__init__"] n2_src_templates_html_templates["html_templates"] n3_ts_migrate_sqlite_to_mariadb["migrate_sqlite_to_mariad"] n4_s_migrate_sqlite_to_postgres["migrate_sqlite_to_postgr"] n5_src_app["app"] n6_src_config["config"] n7_src_dashboard_cache["dashboard_cache"] n8_src_database___init__["__init__"] n9_src_database_access_logs["access_logs"] n1_src_templates___init__ --> n0_rc_templates_template_loader n2_src_templates_html_templates --> n0_rc_templates_template_loader

Findings

LowDependencies declared without a lockfile - non-reproducible builds pyproject.toml
LowSecret-shaped paths present; the code-health audit confirms or clears them img/users-and-secrets.png, src/templates/deception/.config__gcloud__application_default_credentials.json.html, src/templates/deception/.config__polymarket__credentials.json.html

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high4

medium5

low1

Pin third-party GitHub Actions to a commit SHAhigh6 occurrences

.github/workflows

docker/setup-qemu-action@v3, docker/setup-buildx-action@v3, docker/login-action@v3, docker/metadata-action@v5

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Commit a lockfile beside the manifesthigh

pyproject.toml

manifest with no lockfile in scope

An unlocked range means the artifact you tested and the artifact you ship can contain different transitive code, so a malicious patch release reaches production with no diff.

Fix: Run the package manager once and commit the generated lockfile.

Make CI invoke the test suite it hashigh8 occurrences

.github/workflows

8 test files, no test command in the 4 workflows read

A green check that never executed an assertion is worse than no check, because reviewers trust it.

Fix: Add a test step to the existing workflow rather than a new workflow.

Remove continue-on-error from steps that gate correctnesshigh

.github/workflows/pr-checks.yml

line 33 on a correctness step

A failing test suite reports a green check, which is precisely the path this audit exists to close.

Fix: Delete continue-on-error, or move a genuinely advisory step into a separate non-required job.

Enable Dependabot or Renovatemedium

2 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

python:3.13-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Set persist-credentials: false on checkoutmedium

.github/workflows/pr-checks.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Expand the test suite; it cannot cover this codebasemedium

1 test files against 60 source files (ratio 0.017)

A reviewer reads "has tests" and assumes a safety net that covers two files out of eighty-five, which is how untested paths ship behind a green badge.

Fix: Add tests for the highest fan-in modules first.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/docker-build-push.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.