Windows Use

The Problem

Automation of Windows GUI tasks still relies on brittle screen‑scraping or manual scripting. Teams that need a reliable way to let a language model drive applications—click, type, switch windows, or run PowerShell—lack a reusable, API‑driven agent that works directly on the Windows UI Automation tree.

What This Does

windows-use ships an AI‑driven agent that talks to the Windows UI Automation API, interprets LLM output, and performs GUI actions without any computer‑vision model. Core logic lives in windows_use/agent/ (desktop, tree, and service layers) and in the provider package under windows_use/providers/ (e.g., openai, anthropic, google, ollama). The CLI (windows_use/cli/__main__.py) exposes a single entry point (windows-use) that parses --model, --provider, --max-steps, and --debug flags and hands the request to windows_use.Agent.

How It Is Wired

Execution starts at windows_use/cli/__main__.py (invoked by the console script windows-use). The script builds a provider instance (e.g., ChatOpenAI from windows_use/providers/openai/__init__.py) and constructs an Agent (windows_use/agent/__init__.py).

Agent.__init__ stores the LLM client and a Browser enum; Agent.invoke (or ainvoke) forwards the user task to windows_use/agent/desktop/service.py, the highest‑traffic module (24 incoming imports). That service orchestrates:

1. Screen capture & UI tree extraction via windows_use/uia/core.py (largest file, 2158 lines, 24 imports).
2. LLM prompting through the provider’s chat method.
3. Action dispatch to helper modules (windows_use/agent/tools/, windows_use/agent/events/) that call Win32 APIs or PowerShell.

The most‑connected hub is windows_use/providers/base.py (24 incoming, 4 outgoing imports, instability 0.14). It defines the abstract provider interface used by all concrete LLM clients. Circular imports involve windows_use/speech/__init__.py, windows_use/agent/desktop/service.py, and windows_use/providers/openai/__init__.py; breaking these cycles would reduce the 23 detected circular dependencies.

File‑level responsibilities:

Because the hub modules (providers/base, messages/__init__) are imported by many others, changes there have a wide blast radius. The import cycles and deep nesting increase cognitive load for future contributors.

How To Use It

# Install (Python 3.10+ on Windows)
pip install windows-use          # or: uv add windows-use

Set up an LLM key in the environment as described by the provider’s README (e.g., OPENAI_API_KEY for OpenAI).

Run the interactive CLI:

windows-use --provider openai --model gpt-4o --max-steps 150

Or embed the agent in Python code:

from windows_use.providers.openai import ChatOpenAI
from windows_use import Agent, Browser

llm = ChatOpenAI(model="gpt-4o")
agent = Agent(llm=llm, browser=Browser.CHROME)
result = agent.invoke(task="Open Notepad and type a short poem")
print(result.content)

Async usage follows the same pattern with await agent.ainvoke(...) as shown in the README.

Real‑World Use

A support desk could deploy windows-use on a Windows workstation to automate repetitive ticket‑resolution steps: opening the ticketing web UI, copying fields from the UI Automation tree, launching PowerShell scripts, and confirming dialogs—all driven by a single LLM prompt. The agent’s persistent memory lets it maintain context across multiple actions.

Code Health & Issues

• High – GitHub Actions not pinned to commit SHA (.github/workflows/*).
• Medium – No least‑privilege GITHUB_TOKEN permissions declared.
• Medium – No Dependabot/Renovate configuration.
• Medium – No dependency‑vulnerability scan in CI.
• Low – Jobs lack timeout-minutes.

Static analysis also reported:

• Deep nesting (max indentation depth 10) in windows_use/agent/desktop/service.py, windows_use/uia/core.py, windows_use/agent/tree/service.py.
• Import cycles involving windows_use/speech/__init__.py, windows_use/agent/desktop/service.py, windows_use/providers/openai/__init__.py.
• Oversized files (windows_use/uia/core.py, windows_use/uia/enums.py, windows_use/agent/desktop/service.py).
• Broad except: clauses swallowing errors in several service modules.
• Hub module (windows_use/providers/base.py) with 24 dependents; changes here affect many parts.
• File opens without context managers in two modules.
• Repeated code blocks across test and source files.

No lockfile is present; pyproject.toml declares dependencies without reproducible version pins, making builds non‑deterministic.

The Bottom Line

windows-use provides a functional, LLM‑driven Windows GUI automation layer with a clear CLI and Python API, but the codebase suffers from deep nesting, circular imports, and several CI hygiene gaps. It is suitable for teams comfortable navigating a moderately complex Python project and willing to address the highlighted health issues before production deployment.