OpenGuider

Abstract

An AI companion that lives on your desktop - OpenGuider watches your screen, listens to your voice, and guides every step with clear actions.

The Problem

Desktop users often need step‑by‑step guidance for repetitive UI tasks, but existing assistants either stay in a browser tab or require manual scripting. The gap is a locally‑running AI that can watch the screen, listen to voice, and issue precise pointer actions without leaving the desktop.

What This Does

OpenGuider is an Electron‑based desktop AI companion. The core lives in main.js (the Electron main process) and the UI panel lives under renderer/.

The AI logic resides in src/ai/index.js and the orchestration chain in src/agent/*.js.
Screen capture, pointer overlay, and voice I/O are implemented in src/plugins/browser/ (Python side) and src/perception/ui‑scanner.js.
Plugins (e.g., the Browser plugin) expose capabilities through src/plugins/browser/index.js.

Key files:

File	Role
`main.js`	Starts Electron, registers IPC, defines helpers like `resizeCursorOverlayToVirtualBounds`, `debugLog`, and the high‑level command dispatcher (`runPlanShortcutAction`).
`renderer/js/panel/bootstrap.js`	Initializes the panel UI, creates a logger, and calls `init` (entry point `init` → 86 functions).
`src/agent/task-orchestrator.js`	Normalises execution mode, builds the plan, and provides `getSnapshot` used by the UI.
`src/agent/llm-client.js`	Wraps calls to external LLM providers (OpenAI, Claude, etc.).
`src/plugins/browser/python/agent_server.py`	Python side that serves model inference and browser automation.

How It Is Wired

Execution begins when Electron loads main.js. The init function in renderer/js/panel/bootstrap.js (line 523) is invoked via IPC and reaches 86 downstream functions, establishing the UI panel.

From main.js the most travelled paths are:

runPlanShortcutAction → handleOrchestratorResult → speakAssistantResponse → speakText (calls external TTS endpoint).
runPlanShortcutAction → progress (10 calls) → logPostLayer (10 calls) → log (34 callers overall).

The internal call graph shows log, getSnapshot, emitUpdate, on, and addMessage as high‑blast‑radius functions (called ≥10 distinct places). Changing any of these will likely affect many modules.

External interactions are limited to:

Model inference in src/agent/llm-client.js → HTTP request to the provider configured in settings (settings.openaiTtsBaseUrl or similar).
Filesystem reads in src/plugins/browser/sidecar.js (fs.readFileSync for runtime info).
Python plugin src/plugins/browser/python/agent_server.py which invokes api.invoke("get-ollama-models") to discover local models.

No database or persistent storage is used; state is kept in‑memory (src/session/session-manager.js) and persisted only via optional session export.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/OpenGuider
cd OpenGuider

# Install Node dependencies
npm ci        # respects package-lock.json

# Install the Python helper (if you need the browser plugin)
pip install -r src/plugins/browser/python/requirements.txt

# Start the Electron app (the package.json defines the start script)
npm start

Configuration – The UI reads renderer/js/panel/state.js for user settings (model provider, TTS endpoint, etc.). API keys are not shipped; the code expects environment variables such as OPENAI_API_KEY or ANTHROPIC_API_KEY when the corresponding provider is selected.

Running – The desktop UI appears; voice input is enabled if src/perception/voice-listener.js is loaded (requires a microphone). Actions are triggered through the panel or hotkeys registered in renderer/js/panel/bootstrap.js.

Real‑World Use

A support engineer could launch OpenGuider, set the LLM provider to a local Ollama model for privacy, and ask “Set my Outlook signature to the new template”. The AI captures the Outlook window, generates a step plan, and the plugin moves the cursor to the signature field, fills it, and confirms completion—all without leaving the desktop.

Code Health & Issues

HIGH – Pin GitHub Actions – .github/workflows/*.yml use tag references (@v4, @v2). Replace with commit SHA to avoid supply‑chain risk.
HIGH – Remove continue-on-error – release-build.yml masks test failures; delete or move to a non‑gate job.
MEDIUM – Least‑privilege GITHUB_TOKEN – multi-platform-test.yml lacks a permissions block; add contents: read.
MEDIUM – Enable Dependabot – No dependency‑update bot; add .github/dependabot.yml.
MEDIUM – Add vulnerability scan – No dependency‑review step; insert dependency-review-action or osv-scanner.
MEDIUM – Move large binaries to LFS – tutorial.gif (13 MB) and eng.traineddata (5 MB) inflate repo size; track with Git LFS or external storage.
MEDIUM – Checkout without persisting credentials – Set with: persist-credentials: false in multi-platform-test.yml.
LOW – Job timeout – deploy-landing.yml lacks timeout-minutes; add a reasonable bound (e.g., 15).

Additional findings from static analysis: duplicated 6‑line code blocks across the renderer panel files, empty catch {} blocks in three modules, and several oversized files (main.js, renderer/js/panel/ui.js, src/agent/task-orchestrator.js) that would benefit from refactoring.

The Bottom Line

OpenGuider delivers a functional desktop AI assistant with a clear Electron‑based architecture and a modular plugin system. The codebase is usable but carries technical debt (large monoliths, duplicated UI logic) and several CI/security hardening gaps that should be addressed before production deployment. It is a solid foundation for teams needing on‑premise AI guidance, provided they allocate effort to refactor hot spots and tighten the CI pipeline.