HeadlessX

Abstract

The undetected self-hosted browser automation platform. Powered by Camoufox (Firefox) for 0% detection rates. Built for speed, privacy, and scalability.

The Problem

HeadlessX addresses the operational gap between headless browser automation and production reliability: scraping at scale requires undetected browsing, queue-backed job management, and API access control, all self-hosted. Most teams stitch together Playwright scripts, proxy management, and ad-hoc schedulers, which breaks under load and leaves no audit trail.

What This Does

HeadlessX is a self-hosted scraping platform with a web dashboard (apps/web), a protected API (apps/api), and a CLI (packages/cli). It wraps Camoufox (a Firefox fork) via packages/headfox-js to run undetected browser sessions, then exposes operators for website extraction, Google AI Search, Tavily, Exa, and YouTube through apps/api/src/controllers/. A queue system (apps/api/src/services/queue/QueueWorker.ts) backs long-running jobs, and an MCP endpoint (apps/api/src/mcp/server.ts) exposes the API to AI agents.

The platform is a monorepo of 4 self-contained projects: apps/ (web, API, Go HTML-to-MD service, yt-engine), packages/ (headfox-js, CLI), skills/, and scripts/. It uses pnpm, Docker, and Prisma with a PostgreSQL schema in apps/api/prisma/schema.prisma.

How It Is Wired

Execution starts at several entry points. The API server (apps/api/src/app.ts) wires controllers to routes; the MCP server (apps/api/src/mcp/server.ts) handles agent requests via handleMcpRequest, which resolves API keys through resolveMcpApiKey → authenticate (a Prisma DB read) in 3 hops. The headfox-js server (packages/headfox-js/src/server.ts) launches the browser via launchHeadfoxServer → launchOptions → publicIP (an outbound network call via impit.fetch) in 3 hops. The CLI (packages/cli/src/index.ts) is the least connected module (imports 15, imported by 0), a pure entry point.

The highest-blast-radius functions: cn (42 callers, in apps/web/src/lib/utils.ts), writeStructured (25), and requestJson (19). The most-connected modules are apps/web/src/components/playground/shared/index.ts (23 dependents) and apps/api/src/database/client.ts (14 dependents, 0 imports — a stable hub). Five modules sit in an import cycle, including packages/headfox-js/src/utils.ts, which is also an oversized 653-line file. The cn utility being called from 42 places means a signature change ripples across the entire web frontend.

How To Use It

git clone https://github.com/moses-y/HeadlessX
cd HeadlessX
pnpm install
cp .env.example .env  # configure DATABASE_URL, API keys
pnpm --filter @headlessx/api dev  # starts API server
pnpm --filter @headlessx/web dev  # starts dashboard

Configuration lives in .env.example (root) and apps/api/.env (via apps/api/src/env.ts). Google AI Search requires a one-time "Build Cookies" run in the dashboard. The CLI (packages/cli) offers headlessx init, start, logs, status, and doctor. Docker deployment scaffolding is under infra/domain-setup with Caddy.

Real-World Use

A team runs HeadlessX to extract competitor pricing data daily. They hit the API with an API key (POST /api/scrape), which enqueues a job, the queue worker (QueueWorker.ts) picks it up, spins a Camoufox browser via BrowserService.ts, and streams results back through StreamingScrapeController.ts. A downstream agent calls the MCP endpoint to query the same data without a browser.

Code Health & Issues

Static analysis found 114 issues (57 high, 57 medium) across 6 kinds. The most significant:

High – Deep nesting (x44) – e.g., apps/api/src/services/scrape/StreamingScraperService.ts has max indentation depth 9; control flow is hard to follow. Fix: early returns/guard clauses.
High – Import cycle members (x5) – packages/headfox-js/src/utils.ts, StreamingScraperService.ts, JobManager.ts participate in circular imports; change one and you risk runtime initialization order bugs.
High – Duplicated code blocks – 450 repeated 6-line blocks across 88 files, including ExaController.ts, TavilyController.ts, YoutubeController.ts. Fix: extract shared helpers.
Medium – Hub modules (x6) – apps/web/src/components/playground/shared/index.ts (23 dependents) and apps/api/src/database/client.ts (14 dependents) are high-blast-radius churn points.
Medium – Oversized file – packages/headfox-js/src/utils.ts at 653 lines.

SDLC gaps: no CI pipeline (283 source files, zero workflows), no Dependabot/Renovate, the Go Dockerfile uses mutable base images (golang:1.26-alpine), and assets/demo.gif is 86.5MB — should be LFS or external. Test ratio is 0.021 (6 test files vs 283 source files).

The Bottom Line

This is a genuinely functional scraping platform with a real queue, MCP support, and a working CLI — not a scaffold. The codebase is broad but shallow in testing, and the import cycles and duplicated controllers will make maintenance expensive. Use it if you need a self-hosted, undetected scraping stack and can invest in CI and refactoring; skip it if you need something battle-tested out of the box.