The Problem Running ad‑hoc data‑extraction or automation against modern web sites typically requires a custom script per site, manual handling of login cookies, and brittle HTML parsing. Teams spend disproportionate time writing one‑off scrapers that break on UI changes.

What This Does opencli turns any supported web site into a ready‑made CLI command. The core runtime lives in TypeScript under src/.

  • src/main.ts is the binary entry point (exposed as the opencli npm bin). It loads the command registry (src/registry.ts), parses the CLI arguments, and dispatches to the selected command.
  • Each command lives in src/clis/… (e.g., src/clis/twitter/search.ts, src/clis/bilibili/feed.ts). Commands are auto‑registered by src/registry.ts and share a common shape defined in src/types.ts.
  • When a command needs a browser, it calls the dual‑engine API in src/browser/. src/browser/index.ts creates a Playwright MCP bridge, src/browser/page.ts provides goto, evaluate, snapshot, and src/browser/mcp.ts handles the low‑level JSON‑RPC channel.
  • Data extraction pipelines are expressed in YAML files under the same clis/ folders and executed by the pipeline engine (src/pipeline/*). The engine resolves templates (src/pipeline/template.ts), runs steps (src/pipeline/steps/*), and finally renders output (src/output.ts).

How It Is Wired

  1. CLI start – src/main.ts → cli function in src/registry.ts (5 functions, 5 types).
  2. Command dispatch – registry.ts looks up the command name, then calls the exported handler (e.g., search in src/clis/twitter/search.ts).
  3. Browser interaction – Handlers import src/browser/index.ts; the most‑used functions are render, evaluate, wait, goto (called 17–18 times each across the code base).
  4. Pipeline execution – src/pipeline/executor.ts orchestrates steps; it calls render (5 × calls), evaluate (60 ×), and wait (59 ×).
  5. Output – Results flow into src/output.ts (7 functions, 1 class) which supports JSON, Markdown, CSV, etc.

The hub modules src/registry.ts (64 inbound imports) and src/types.ts (30 inbound imports) have the largest blast radius; any change here ripples through the entire command set. No circular import cycles were detected, so the graph is stable.

How To Use It

# Clone and install
git clone https://github.com/moses-y/opencli
cd opencli
npm ci          # installs exact versions from package-lock.json

# One‑time browser token discovery
npx opencli setup   # runs src/setup.ts → src/browser/index.ts → MCP bridge

# List available commands
npx opencli list

# Example: fetch the latest tweets
npx opencli twitter search --query "openai"

All configuration lives in the user's Chrome profile; the tool reads the PLAYWRIGHT_MCP_EXTENSION_TOKEN automatically during setup. No additional environment files are required.

Real‑World Use A product team can add a new site by dropping a .ts or .yaml adapter into src/clis/. After opencli setup the command is instantly callable, and the doctor command (src/doctor.ts) can verify token validity across all adapters, reducing onboarding time from days to minutes.

Code Health & Issues

  • High – Hub modules (src/registry.ts, src/types.ts) concentrate 94 % of imports; keep them minimal and backward‑compatible.
  • High – Duplicated 6‑line blocks across 32 files (e.g., scripts/postinstall.js, src/completion.ts); extract into shared helpers.
  • Medium – Empty catch {} blocks in src/interceptor.ts, src/clis/barchart/flow.ts, etc.; add logging or re‑throw.
  • Medium – High branching density (92 branches/131 lines) in src/pipeline/template.ts, src/browser/errors.ts; consider strategy tables.
  • Medium – Deep nesting (max depth 6) in several CLI adapters; refactor with early returns.

Health‑audit findings

  • HIGH – Pin GitHub Action versions to commit SHAs (.github/workflows/*).
  • MEDIUM – Declare least‑privilege GITHUB_TOKEN permissions (.github/workflows/ci.yml).
  • MEDIUM – Add Dependabot (.github/dependabot.yml).
  • MEDIUM – Include a dependency‑vulnerability scan step in CI.
  • MEDIUM – Set persist-credentials: false on checkout steps.
  • MEDIUM – Review/disable the postinstall script in package.json.
  • LOW – Define timeout-minutes on workflow jobs.
  • LOW – Add repository conventions (.editorconfig, formatter config).

The Bottom Line opencli delivers a functional, AI‑ready CLI layer over browsers with a clear separation between command registration, browser control, and pipeline rendering. The codebase is well‑tested and CI‑covered, but the central registry and type modules are high‑impact change zones, and several maintainability hotspots (duplicate code, empty catches, deep nesting) should be addressed before large‑scale extensions. It is a solid foundation for teams that need rapid, reproducible web data access without writing bespoke scrapers.