WaterCrawl

The Problem

Enterprises that need to feed large‑language‑model pipelines with fresh web content must build a custom crawler, storage layer, and API. Doing this from scratch involves stitching together Scrapy, Celery, a Django REST API, and a file store – a high‑maintenance effort that delays product rollout.

What This Does

WaterCrawl delivers a ready‑made stack that crawls sites, stores results in MinIO, and exposes a documented OpenAPI endpoint. The backend lives under backend/ and is a Django project (backend/watercrawl/). Core crawling logic lives in backend/spider/ (Scrapy spiders) and asynchronous processing is handled by Celery (backend/watercrawl/celery.py). The frontend UI is a React/Next‑style app in frontend/, but the API can be used head‑less.

Key entry points:

• backend/manage.py – Django’s CLI; runserver starts the API.
• backend/watercrawl/asgi.py / wsgi.py – entry points for ASGI servers (e.g., Daphne, Gunicorn).
• backend/watercrawl/celery.py – creates the Celery app used by workers.

The API routes (backend/watercrawl/urls.py) delegate to app‑specific routers (backend/core/urls.py, backend/user/urls.py, etc.). Each view (e.g., backend/core/views.py) calls a service layer (backend/core/services.py) which may enqueue a Celery task (backend/core/tasks.py). The task launches a Scrapy crawl via backend/spider/spiders/* and streams results back through Django models (backend/core/models.py).

How It Is Wired

1. Startup – docker/docker-compose.yml builds two containers: backend (from backend/Dockerfile) and nginx. The backend container runs entrypoint.sh → manage.py migrate && gunicorn backend.wsgi:application.
2. Request Flow – An HTTP POST to /api/crawl/ hits backend/watercrawl/urls.py → backend/core/views.CrawlCreateView.
3. Service Layer – CrawlCreateView calls backend/core/services.create_crawl. This writes a CrawlRequest DB row (backend/core/models.CrawlRequest) and calls backend/core/tasks.run_crawl.delay(request_id).
4. Celery Worker – The worker process loads backend/watercrawl/celery.py, picks up the task, and executes backend/spider/spiders/scraper.py via Scrapy’s CrawlerProcess.
5. Result Persistence – Scrapy pipelines (backend/spider/pipelines.py) upload extracted files to MinIO using credentials from backend/.env.example (e.g., MINIO_SERVER_URL). The pipeline also creates CrawlResult model entries (backend/core/models.CrawlResult).
6. Feedback – The task updates the CrawlRequest status; the API endpoint streams SSE updates from backend/common/views.StreamView.

Ownership map

The graph is a clear star: manage.py → URL router → view → service → task → spider. No circular imports are evident, so modifying any layer has limited blast radius.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/WaterCrawl.git
cd WaterCrawl

# Build and start containers (Docker compose)
cd docker
cp .env.example .env          # edit MINIO_* and DB vars as needed
docker compose up -d          # starts backend, nginx, redis, celery, minio

# Apply migrations and start the API (if not using compose entrypoint)
docker exec -it watercrawl_backend bash
python manage.py migrate
gunicorn backend.wsgi:application --bind 0.0.0.0:8000

Configuration: Required variables live in backend/.env.example (database URL, MINIO_SERVER_URL, MINIO_ACCESS_KEY, MINIO_SECRET_KEY). The same file is copied into the Docker compose environment.

Running a crawl:

curl -X POST http://localhost/api/crawl/ \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "depth": 2}'

The response includes a request_id; SSE endpoint /api/crawl/{id}/events/ streams progress.

Real‑World Use

A data‑science team can schedule nightly crawls of competitor blogs, store raw HTML in MinIO, and feed the extracted text into a downstream LLM fine‑tuning pipeline. The API call above can be wrapped in an Airflow DAG that polls the SSE endpoint, downloads the result files, and pushes them to a vector store.

Code Health & Issues

• Low – Secret‑shaped path – docker/.env.local contains placeholder secrets; should be excluded from VCS.
• Test suite present (31 test files) but CI only runs backend-tests.yml; coverage of the frontend is missing.
• No license file beyond the generic LICENSE placeholder; verify compliance before commercial use.
• README.md references the upstream repo (watercrawl/watercrawl) – some links may be stale.

The Bottom Line

WaterCrawl offers a complete, Docker‑ready stack for web crawling and LLM data ingestion, with a clean separation between API, async processing, and scraping. The codebase is well‑structured, but production deployments must secure the .env files and verify licensing. It is suitable for teams that need a self‑hosted crawler without building the plumbing from scratch.