The Problem

AI agents are confined to chat boxes and terminals. They can write code and run tests, but cannot click a button, send an email, or fill out a form on a user's Mac. This limits agents to tasks that fit inside a text interface, leaving the entire GUI desktop unreachable.

What This Does

Ghost OS gives AI agents full computer-use on macOS. It observes and operates every app through the accessibility tree and input event taps, without screenshots or a vision model. The core lives in Sources/GhostOS/ as a Swift package, with a Python sidecar (vision-sidecar/server.py) for vision-based perception when needed.

The system records user actions during a task (via CGEvent taps), enriches them with accessibility context, and synthesizes them into replayable recipes. These recipes are stored as JSON in recipes/ (examples: gmail-send.json, slack-send.json) and can be parameterized and re-run by an agent.

How It Is Wired

Execution starts at Sources/ghost/main.swift, which routes through SetupWizard.swift for initial configuration (permissions, API keys) and Doctor.swift for diagnostics. The MCP server (Sources/GhostOS/MCP/MCPServer.swift) exposes tools to agents via MCPTools.swift.

The learning pipeline is the core flow: LearningRecorder.swift captures input events, EventHandlers.swift processes them, LearningDispatch.swift routes them, and AppSwitchDetector.swift tracks context changes. The synthesized recipe flows through RecipeEngine.swift and RecipeStore.swift for persistence.

The vision path is separate: VisionBridge.swift and CDPBridge.swift handle Chrome DevTools Protocol integration, while vision-sidecar/server.py runs a Python HTTP server for vision model inference. This is the only network-facing component.

The import graph shows a single module with zero edges — every file is effectively independent, so there is no hub, no cycle, and no central dependency to worry about. That also means no shared abstraction layer; each file owns its effects locally.

How To Use It

git clone https://github.com/moses-y/ghost-os
cd ghost-os
swift build          # builds the Swift package
swift run ghost setup   # configure permissions and API keys

The vision sidecar requires Python dependencies from vision-sidecar/requirements.txt:

pip install -r vision-sidecar/requirements.txt
python vision-sidecar/server.py   # start the vision server

Requires macOS 14+, Swift 6.2, and Input Monitoring permission (System Settings > Privacy & Security > Input Monitoring). No CI pipeline or lockfile exists, so builds are not reproducible.

Real-World Use

An agent can learn a task by watching a user:

User:    "Watch me send an email."
Agent:   ghost_learn_start task_description:"send email in Gmail"
         ...user performs the task...
Agent:   ghost_learn_stop
         -> 8 actions with full AX context
         -> Synthesizes recipe with 3 parameters: recipient, subject, body
         -> ghost_recipe_save
User:    "Send an email to bob about the Q4 report"
Agent:   ghost_run recipe:"gmail-send-learned" params:{...}

Code Health & Issues

Static analysis found 29 issues (4 high, 25 medium) across 6 kinds:

  • High - Deep nesting (x11) in Actions.swift, Perception.swift, CDPBridge.swift — max indentation depth 8; fix with early returns and guard clauses.
  • High - Duplicated code blocks (x20) across 7 files — extract shared helpers.
  • Medium - File opened without context manager in vision-sidecar/server.py — use with open(...).
  • Medium - Broad exception handling in vision-sidecar/server.py — catch specific exceptions.
  • Medium - Oversized files (x3): Actions.swift (953 lines), Perception.swift, SetupWizard.swift — split by responsibility.
  • Medium - High branching density (x12) — 352 branch points over 953 lines in Actions.swift.

SDLC observations: no CI pipeline (29 source files, zero build gates), no dependency lockfile, and 1 test file against 29 source files (ratio 0.034). A 10.6MB GIF (demo-new-tools.gif) is committed; consider Git LFS. No secrets committed.

The Bottom Line

Ghost OS is a functional computer-use framework with a novel self-learning recipe system, but it is early-stage: no CI, minimal tests, and heavy cognitive load in core files. The architecture is simple enough to extend — each module is independent — but that also means no shared safety net. Worth exploring if you need GUI automation for agents on macOS and can tolerate the maintenance burden.