Here's a concise, professional technical briefing for the crawlee repository, written to the exact specification.

The Problem

This repository provides a Node.js library for web scraping and browser automation, used to build reliable crawlers for data extraction, AI, LLMs, RAG, and GPTs. It supports headless and headful modes via Puppeteer and Playwright, raw HTTP, and CSS selectors via Cheerio and JSDOM, with proxy rotation. The primary pain point for engineers is the circular dependency structure and high cognitive load in the core modules, which makes extending or debugging the system difficult without tracing many interdependent files.

What This Does

Crawlee is a monorepo structured around packages/ and website/. It exports crawler types like PlaywrightCrawler, CheerioCrawler, and BasicCrawler from entry points in packages/*/src/index.ts. The library resolves to 1118 internal modules with 523 import edges and 60 modules inside circular dependencies. Execution typically starts at a crawler entry point, flows through a requestHandler, and fans out to network I/O, storage (Dataset/KeyValueStore/RequestQueue), and browser control. The core module at packages/core/src/index.ts is imported by many packages but imports 17 others, giving it a blast radius of 17. The packages/core/src/configuration.ts hub is depended on by 50 modules; changes here ripple broadly. The test/e2e/tools.mjs file shows max indentation depth of 7, indicating deeply nested control flow that is hard to follow.

How It Is Wired

  • Entry Points: packages/basic-crawler/src/index.ts, packages/browser-crawler/src/index.ts, packages/cheerio-crawler/src/index.ts, packages/browser-pool/src/index.ts, packages/cli/src/index.ts.
  • Call Graph: Execution starts at a crawler constructor, flows through requestHandler, and exits via storage writes or network requests. The internal call graph shows 2 resolved call edges between repository functions.
  • External Effects: 1 function reads or writes files.
  • Key Files:
  • packages/core/src/typedefs.ts – Defines entries, keys, Constructor, Awaitable, AllowedHttpMethods (2 functions, 3 classes/types).
  • packages/types/src/utility-types.ts – Defines Dictionary, Constructor, Awaitable, AllowedHttpMethods (4 classes/types).
  • packages/memory-storage/src/fs/key-value-store/memory.ts – Defines get, update, delete, KeyValueMemoryEntry.
  • packages/core/src/request.ts – High branching density (24 branch points over 65 lines); participates in circular imports.
  • packages/core/src/configuration.ts – Hub module; 50 dependents; participates in circular imports.
  • packages/browser-pool/src/playwright/playwright-plugin.ts – Circular dependency; Ca 7, Ce 11, instability 0.61.

How To Use It

  • Setup: npm install crawlee playwright (or yarn add crawlee playwright). The CLI scaffolds projects via npx crawlee create my-crawler.
  • Configuration: No environment variables are strictly required for basic operation, but proxy rotation and browser binaries require separate setup. The package.json at root and per-package level manage dependencies.
  • Running It: npm start from a scaffolded project, or invoke packages/cli/src/index.ts directly. The README documents npx crawlee create my-crawler followed by npm start.

Real-World Use

A developer uses PlaywrightCrawler to render JavaScript-heavy sites. They define an async requestHandler({ request, page }) that extracts data via page.locator() or Cheerio, then pushes results with Dataset.pushData(). The crawler queues new URLs via enqueueLinks(). Data lands in ./storage/datasets/default. Proxy configuration is set at instantiation via proxyConfiguration: { proxyUrls: [...] }.

Code Health & Issues

The static analysis (209 total: 94 high, 106 medium, 9 low) found the following production-impacting findings:

  • [HIGH] Pin third-party GitHub Actions to commit SHA – .github/workflows: EndBug/add-and-commit@v9, nick-fields/retry@v3, styfle/cancel-workflow-action@0.13.0, apify/workflows/execute-workflow@main. A tag can move, meaning CI runs with whatever the owner last pushed, risking secret exposure.
  • [HIGH] Open a pull request instead of pushing to the default branch – .github/workflows/release.yml: git push lands on the deploying branch with no tests having run against the result.
  • [MEDIUM] Declare least-privilege permissions for GITHUB_TOKEN – .github/workflows/deploy-nginx.yml: 3 workflows declare no permissions; the token inherits repository defaults, allowing injected steps to push or mint releases.
  • [MEDIUM] Install from the lockfile in CI – .github/workflows: yarn install without --immutable means a fresh resolution in CI may differ from the locked set.
  • [MEDIUM] Pin the container base image by digest – packages/templates/templates/camoufox-ts/Dockerfile: apify/actor-node-playwright-chrome:20-1.50.1 is mutable; today's build and last month's contain different libc and CVEs.
  • [MEDIUM] Gate pull requests on a dependency vulnerability scan – .github/workflows: No dependency scan is currently in CI.
  • [MEDIUM] Move large binaries to Git LFS – website/versioned_docs/version-3.15/api-typedoc.json: 8 blobs over 5MB (api-typedoc.json 11.2MB, 11.1MB, 10.8MB).
  • [MEDIUM] Set persist-credentials: false on checkout – .github/workflows/publish-to-npm.yml: checkout keeps the token; a malicious postinstall script could read pushable credentials.
  • [LOW] Set timeout-minutes on workflow jobs – .github/workflows/deploy-nginx.yml: 4 jobs declare no timeout, risking six-hour platform defaults overlapping on a two-hourly schedule.

The Bottom Line

Crawlee is a capable, well-structured library for Node.js web scraping and browser automation, particularly valuable for AI/ML data ingestion. The monorepo organization and typed APIs are strengths. However, the circular dependency cycle in core modules (configuration.ts, request.ts, browser-controller.ts) and the high blast radius of the configuration hub create a moderate barrier to entry for contributors. The CI hygiene issues (mutable GitHub Action SHAs, missing lockfile installs, unpinned Docker bases) are actionable and should be addressed before onboarding new teams. Use it if you need a typed, crawler-ready framework; avoid if you require a lightweight, zero-config HTTP scraper.