clacky

The Problem

Windows users have no open-source equivalent to Clicky, the macOS AI desktop companion that sees your screen and acts on it. Existing Windows automation tools require scripting or lack the voice-driven, screen-aware interaction model. Clacky fills that gap with a voice-first companion that can see, point at, and operate a Windows desktop.

What This Does

Clacky is a voice-first desktop companion for Windows. Hold a hotkey, speak, and it answers questions about your screen, points at UI elements with an on-screen buddy, and performs actions like opening apps, clicking, and typing using Claude Computer Use. It maintains cross-session memory and can learn voice-taught routines.

The repo also contains a separate file organizer (clacky organize) that tidies folders via LLM calls with reversible moves (clacky undo). A third project, organizer/, appears to be a standalone version of this functionality.

How It Is Wired

Execution starts at clacky/cli.py (main, _cmd_run, _cmd_organize). The main entry reaches 40 functions; _cmd_run reaches 42. The primary voice companion lives in clacky/shell/, with companion_manager.py as the hub—79 functions, called from 6 files, calling into 40 others. It owns the core loop: _ensure_ollama_running, _build_system_prompt, and session orchestration.

The clacky/agent/ package handles the agentic core: actuation.py (23 functions, 3 classes) provides left_click, double_click, type_text, and other mouse/keyboard actions. permission.py gates actions with 7 importing modules. The providers/ package abstracts LLM backends—base.py is the most-imported module (Ca 10), while factory.py (instability 0.88) is a thin dispatcher.

The shell subproject handles UI (ui/overlay.py, ui/panel.py), audio (audio/ambient_listener.py), and AI providers (ai/ with Claude, Gemini, OpenAI, Ollama, and GitHub Copilot implementations). web_search.py's search function is called from 17 places—the widest blast radius. slog (15 callers) and show_notification (13 callers) are similarly central.

Traced external effects: main → maybe_show_setup_wizard → mark_setup_complete writes a flag file; _worker → device_login makes an HTTP POST and opens a browser. The code touches the filesystem (48 functions), network (24), model inference (6), external commands (4), and databases (1).

How To Use It

Setup (Python 3.10+, Windows 10/11):

git clone https://github.com/moses-y/clacky
cd clacky
pip install -e ".[shell,claude]"

Configuration: Copy .env.example → clacky/shell/.env with ANTHROPIC_API_KEY, DEEPGRAM_API_KEY, CLACKY_ACTIVE_LLM=claude.

Running:

clacky run        # voice companion
clacky organize ~/Desktop --dry-run   # file organizer, preview

Real-World Use

A user holds Ctrl+Alt+M, says "open Notepad and type hello." The companion captures audio, transcribes via Deepgram, routes to Claude, resolves the target UI element via element_locator.py, and executes type_text from actuation.py. The on-screen buddy points at Notepad during the action.

Code Health & Issues

Static analysis found 69 issues (11 high, 57 medium, 1 low). High-severity findings:

• Import cycle (3 files): organizer/__init__.py, planner.py, executor.py are mutually reachable.
• Duplicated code: 22 repeated 6-line blocks across 16 files.
• Deep nesting (15 files): shell/actions.py reaches indentation depth 13.
• Oversized file: shell/companion_manager.py has 1253 lines.

Medium: 35 broad exception handlers, 5 files opened without context managers. No CI/CD, no lockfile, no Dependabot. Tests exist (9 files) but nothing runs them automatically.

The Bottom Line

Clacky is a functional, ambitious prototype that works for its core demo loop. The codebase is rough—deep nesting, duplication, and a monolithic manager file will make changes expensive. Worth using for the voice-desktop-companion pattern on Windows, but expect to refactor before building on it.