The Problem
AI agents can write code and search files, but they operate inside a chat box. They cannot click buttons, send emails, or fill out forms in native macOS applications. Existing solutions rely on brittle screenshot-based computer vision, which is slow, expensive, and breaks when the UI changes. Ghost OS replaces that with direct accessibility-tree access and system-level event observation, giving agents native control of the Mac without a vision model.
What This Does
Ghost OS is a Swift 6.2 package that exposes macOS computer-use capabilities to AI agents through the Model Context Protocol (MCP). The core dispatch lives in Sources/GhostOS/MCP/MCPServer.swift and MCPTools.swift, which register tools for actions, perception, and recipes. Actions like clicking, typing, and dragging are implemented in Sources/GhostOS/Actions/Actions.swift using the macOS accessibility API, coordinated by FocusManager.swift.
The distinctive feature is self-learning workflows. Sources/GhostOS/Learning/LearningRecorder.swift captures user actions through a CGEvent tap enriched with accessibility-tree context. LearningDispatch.swift and AppSwitchDetector.swift process those events, and RecipeEngine.swift synthesizes them into parameterized, replayable recipes stored as JSON in recipes/. The vision-sidecar/server.py provides an optional Python service for vision-based annotation, but the core path works without it.
How To Use It
Setup: Build the Swift package with swift build (uses Package.swift and Package.resolved). The Python sidecar requires pip install -r vision-sidecar/requirements.txt. Run ghost setup (implemented in Sources/ghost/SetupWizard.swift) to configure permissions.
Configuration: Requires Input Monitoring permission (System Settings > Privacy & Security > Input Monitoring) for the learning features. No environment variables are documented.
Running it: The CLI entry point is Sources/ghost/main.swift. The README does not document a specific run command, so you would invoke the built binary (e.g., swift run ghost) or start the MCP server through your agent's MCP client. The vision sidecar runs via python vision-sidecar/server.py.
Real-World Use
A workflow: the user records a Gmail send once, Ghost OS observes and synthesizes a recipe, and the agent replays it later:
ghostlearnstart taskdescription:"send email in Gmail" ...user performs the task manually... ghostlearnstop -> recipe synthesized with parameters: recipient, subject, body ghostrecipesave ghostrun recipe:"gmail-send-learned" params:{recipient:"bob@example.com", subject:"Q4 report", body:"..."}
Code Health & Issues
Med - No CI/CD pipeline - no .github/ or CI config; no automated build or test gate. Tests exist (Tests/GhostOSTests/LocatorBuilderTests.swift) but only cover one file. Low - No dependency lockfile - vision-sidecar/requirements.txt declares dependencies without pins, so builds are not reproducible. Low - Untested learning pipeline - the core LearningRecorder, AppSwitchDetector, and RecipeEngine have no test coverage; these are the highest-risk components. Low - Single-file test coverage - only LocatorBuilderTests.swift is tested; the MCP server, actions, and perception layers are untested.
The codebase is well-organized with clear separation (actions, perception, learning, recipes, MCP). Documentation is solid (README.md, docs/progress.md, GHOST-MCP.md). The MIT license is present.
The Bottom Line
Ghost OS is a technically interesting approach to computer-use that avoids the cost and fragility of vision models. The self-learning recipe system is the differentiator, but it is the least-tested part of the codebase. This is suitable for developers building macOS automation for AI agents who want a native, accessibility-based alternative to screenshot approaches; production teams should add CI and test coverage before relying on it.