pilot shell - Moses Yebei

Abstract

How real engineers run Claude Code and Codex: spec-driven planning, enforced TDD, persistent memory, and quality enforcement on all levels. Make your agents production-ready.

The Problem

Engineers using Claude Code or Codex get fast code generation but lose the discipline of spec‑driven design, test‑first development, and reproducible knowledge. The result is fragmented implementations, missing tests, and drifting requirements that make production hand‑off risky.

What This Does

pilot-shell bundles five self‑contained projects that together impose a production‑grade workflow on LLM‑generated code:

console/ – a TypeScript web dashboard and CLI that orchestrates the LLM, runs spec creation (/spec), TDD (/fix), and continuous quality gates. Core entry points are console/src/cli/adapters/index.ts and console/src/cli/handlers/index.ts.
pilot/ – the Python‑backed knowledge engine that stores specifications, memories, and the semantic graph (Semble/CodeGraph). Files such as pilot/hooks/_lib/util.py and pilot/hooks/_checkers/charset.py provide the validation layer.
launcher/ – a thin Node wrapper that launches the dashboard in a Docker container (launcher/ contains only executable TS files).
installer/ – the automated installer that configures the environment, pulls secrets, and seeds the vector store. The hub module installer/context.py is imported by 21 other modules.
scripts/ – one‑off utilities (e.g., scripts/build-hooks.js) used by CI.

All projects share the same Docker dev container (.devcontainer/Dockerfile) and a GitHub Actions CI pipeline (.github/workflows/*). The repo is deliberately split so each component can be versioned or replaced independently.

How It Is Wired

Startup – console/src/cli/adapters/index.ts registers adapters for Claude (claude-code.ts) and Codex (codex.ts). The adapter returns a Session object consumed by the handler layer.
Command handling – console/src/cli/handlers/index.ts dispatches CLI verbs (spec, fix, build) to concrete handlers (session‑init.ts, user‑message.ts). Each handler builds a Context via console/src/services/context/ContextBuilder.ts.
Context generation – ContextBuilder pulls stored specs from the Python pilot (pilot/hooks/_lib/util.py) through an HTTP MCP server (console/src/servers/mcp-server.ts). The server loads the knowledge graph and returns a JSON spec.
Execution loop – The selected handler invokes console/src/services/context/ObservationCompiler.ts, which streams LLM observations to the TokenCalculator for cost‑aware prompting. Results are fed back to the UI (console/src/components/*) and persisted by installer/context.py.
Persistence – installer/context.py (the highest‑degree hub) writes spec snapshots and vector embeddings to the local SQLite store used by the pilot. Because it is imported by 21 modules, any change here has a wide blast radius.
CI/CD – GitHub Actions run npm ci in console/ and pip install -r requirements.txt (implicit in the Python installer) before executing the test suites (tests/ under each project). The workflow also builds the Docker image defined in .devcontainer/Dockerfile.

The import graph shows two small cycles (installer/steps/claude_files.py ↔ installer/steps/pilot_files.py) that increase maintenance friction, and a handful of deep‑nesting hotspots (e.g., installer/steps/dependencies.py).

How To Use It

# Clone the repo
git clone https://github.com/moses-y/pilot-shell
cd pilot-shell

# Build the dev container (Docker required)
docker build -f .devcontainer/Dockerfile -t pilot-shell-dev .

# Install the Node front‑end
cd console
npm ci

# Run the installer (creates .env.example, seeds vector store)
cd ../installer
python -m installer.cli install   # entry point defined in installer/cli.py

# Start the dashboard
cd ../console
npm run dev   # starts Vite dev server on http://localhost:5173

The installer expects a Claude or OpenAI token in docs/site/.env.development (currently committed – see health section). After the dev server is running, the CLI can be invoked directly, e.g.:

node console/src/cli/commands.ts spec new-feature

Real‑World Use

A team adds a new microservice: they run pilot-shell console spec new-service, write a high‑level requirement, and the system scaffolds a TDD suite, generates skeleton code, and stores the spec in the shared vector store. Subsequent developers can query the knowledge graph (pilot/ APIs) to retrieve rationale, reducing onboarding time and preventing duplicate effort.

Code Health & Issues

Critical – Secrets in workflow .github/workflows/claude.yml (CLAUDE_CODE_OAUTH_TOKEN).
High – Tracked .env.development in docs/site/; unpinned GitHub Action (anthropics/claude-code-action@v1); continue‑on‑error in release-dev.yml; direct push to main in release.yml.
Medium – Base image not pinned in .devcontainer/Dockerfile; checkout persists credentials in deploy-website.yml.
Low – No timeout on claude.yml jobs.

Resource safety – open(...) without context manager in installer/ui.py and two pilot hooks.
Resilience – Broad except: clauses in installer/platform_utils.py and related steps.
Cognitive load – Deep nesting (up to 8 levels) and oversized files (installer/steps/dependencies.py ≈ 1.5 k LOC).
Soundness – Circular imports between installer/steps/claude_files.py and installer/steps/pilot_files.py.
Clarity – Hub module installer/context.py is a churn hotspot.
Duplication – Repeated UI snippets across 17 component files in docs/site/src/components/.

All findings stem from deterministic static analysis; no additional issues were inferred.

The Bottom Line

pilot-shell provides a concrete, spec‑driven wrapper around Claude Code/Codex that enforces TDD, persistent memory, and CI checks. The architecture is modular, but the installer and context hub are high‑impact change points, and several security‑related secrets are currently exposed. Teams ready to adopt LLM‑assisted development can benefit, provided they remediate the critical secret leaks and tighten the most tangled modules.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

1,337 files

TypeScript482
Markdown279
Python205
TSX199
JSON43
JavaScript17

ReactDjangoExpressTailwindDockerVector store

Module dependencies

The 10 most depended-upon modules of 906, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_installer_context["context"] n1_installer_ui["ui"] n2_installer_steps_base["base"] n3_installer_platform_utils["platform_utils"] n4_installer_manifest["manifest"] n5_installer___init__["__init__"] n6_installer_claude_paths["claude_paths"] n7_installer_steps_claude_files["claude_files"] n8_installer_steps_dependencies["dependencies"] n9_installer_steps_pilot_files["pilot_files"] n0_installer_context --> n1_installer_ui n2_installer_steps_base --> n0_installer_context n7_installer_steps_claude_files --> n6_installer_claude_paths n7_installer_steps_claude_files --> n0_installer_context n7_installer_steps_claude_files --> n3_installer_platform_utils n7_installer_steps_claude_files --> n2_installer_steps_base n7_installer_steps_claude_files --> n9_installer_steps_pilot_files n8_installer_steps_dependencies --> n6_installer_claude_paths n8_installer_steps_dependencies --> n0_installer_context n8_installer_steps_dependencies --> n4_installer_manifest n8_installer_steps_dependencies --> n3_installer_platform_utils n8_installer_steps_dependencies --> n2_installer_steps_base n9_installer_steps_pilot_files --> n0_installer_context n9_installer_steps_pilot_files --> n3_installer_platform_utils n9_installer_steps_pilot_files --> n2_installer_steps_base n9_installer_steps_pilot_files --> n7_installer_steps_claude_files classDef cyc stroke:#ef4444,stroke-width:2px; class n7_installer_steps_claude_files,n9_installer_steps_pilot_files cyc;

Findings

LowSecret-shaped paths present; the code-health audit confirms or clears them docs/site/.env.development

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

critical1

high5

medium2

low1

Keep secrets out of workflows a fork can triggercritical

.github/workflows/claude.yml

CLAUDE_CODE_OAUTH_TOKEN (publish or cloud scope)

Contributor-controlled input reaches a job holding a publish token or cloud credential, so a pull request becomes credential exfiltration.

Fix: Move secret-using steps into a workflow_run job that never checks out pull request code, or gate on an environment with required reviewers.

Untrack the file your own .gitignore says to ignorehigh

docs/site/.env.development

tracked although .gitignore excludes it

The author already decided this file must never be committed, so its presence is an accident nobody noticed, which means the credentials in it are the ones actually in use.

Fix: git rm --cached the path and rotate; the ignore rule is already correct.

Pin third-party GitHub Actions to a commit SHAhigh

.github/workflows

anthropics/claude-code-action@v1

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Remove the committed .env and rotate what it holdshigh

docs/site/.env.development

A tracked .env is the most common route for a working key to reach a public clone, and it is the file the app actually loads, so the value is usually live.

Fix: git rm --cached the file, add it to .gitignore, rotate every credential it names, commit a .env.example with empty values.

Remove continue-on-error from steps that gate correctnesshigh

.github/workflows/release-dev.yml

line 107 on a correctness step

A failing test suite reports a green check, which is precisely the path this audit exists to close.

Fix: Delete continue-on-error, or move a genuinely advisory step into a separate non-required job.

Open a pull request instead of pushing to the default branchhigh

.github/workflows/release.yml

git push origin main

Automated commits land on the branch that deploys, with no test having run against the result.

Fix: Push to a bot branch and open a pull request, or restrict the push to a tag ref.

Pin the container base image by digestmedium

.devcontainer/Dockerfile

mcr.microsoft.com/devcontainers/base:trixie

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Set persist-credentials: false on checkoutmedium

.github/workflows/deploy-website.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/claude.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.