crawlee

The Problem

Web scraping in Node.js usually means stitching together HTTP clients, browser automation tools, and custom storage code. Teams end up maintaining fragile crawlers that break when sites change, get blocked by bot protections, or lose data when processes crash. There is no standard way to handle retries, proxy rotation, request queues, and result storage across different crawling strategies.

What This Does

Crawlee is a monorepo (managed with lerna.json) that provides a unified crawling framework for Node.js. The packages/ directory contains modular crawler implementations — basic-crawler, cheerio-crawler, playwright-crawler, puppeteer-crawler, jsdom-crawler, and http-crawler — each handling a different scraping approach. Core services like request queueing, dataset storage, and session management are shared across all crawler types.

The docs/ directory (173 files) is the real deliverable here. It includes a full introduction tutorial (docs/introduction/), quick-start guides (docs/quick-start/), and operational guides covering proxy management (docs/guides/proxymanagement.mdx), session handling (docs/guides/sessionmanagement.mdx), scaling (docs/guides/scalingcrawlers.mdx), and anti-blocking techniques (docs/guides/avoidblocking.mdx). Each guide ships with working TypeScript examples, not just prose.

The package exposes a single crawlee NPM package that re-exports the modular components. This is a fork of apify/crawlee (25k+ stars), so the codebase is mature and battle-tested.

How To Use It

Setup: Install via the crawlee NPM package. For browser automation, you need to install Playwright or Puppeteer separately — they are not bundled to keep install size down.

npm install crawlee playwright

Configuration: No environment variables or config files are required for local development. Storage defaults to ./storage/datasets/default on disk. Proxy configuration is done programmatically in the crawler options, as shown in docs/guides/proxymanagement.mdx.

Running it: The entry point is your own script. The quickest start is the CLI generator:

npx crawlee create my-crawler cd my-crawler npm start

For manual setup, instantiate a crawler class and define a requestHandler. The basic pattern is shown in docs/quick-start/quickstartcheerio.ts:

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({ async requestHandler({ request, $, enqueueLinks }) { await Dataset.pushData({ title: $('title').text(), url: request.loadedUrl }); await enqueueLinks(); }, });

await crawler.run(['https://crawlee.dev']);

Real-World Use

A common production pattern is scraping an e-commerce site with anti-bot protection. Use PlaywrightCrawler with the fingerprint evasion techniques from docs/guides/avoidblockingplaywright.ts, route requests through a rotating proxy pool from docs/guides/proxymanagementintegrationplaywright.ts, and persist results to a dataset for downstream ETL into a database or vector store for RAG pipelines.

Code Health & Issues

The repository is well-structured for a mature open-source project. CI is comprehensive — GitHub Actions covers unit tests (test-ci.yml), end-to-end tests (test-e2e.yml), npm publishing (publish-to-npm.yml), and PR title validation (check-pr-title.yml). A LICENSE.md, CHANGELOG.md, and MIGRATIONS.md are present. The docs/ directory doubles as the test suite — every example file is a runnable TypeScript script, which is a strong pattern for keeping docs honest. Low - Monorepo complexity: lerna.json with multiple packages means a steeper contribution learning curve, though it's standard for this scale. Low - No visible security audit config (e.g., npm audit in CI, Dependabot) in the workflow files listed. Low - The docs/ folder mixes .mdx prose with .ts examples; the coupling between them isn't enforced by tooling, so examples could drift from the text over time.

The Bottom Line

Crawlee is a production-grade scraping framework with excellent documentation and a proven track record via its upstream project. It's the right choice if you need consistent crawling infrastructure across HTTP, Cheerio, and headless browser strategies without building it yourself. The main cost is the learning curve of its abstractions; for simple one-off scrapes, a plain fetch loop is still simpler.