The Problem

Users need a way to drive Android devices or a desktop computer through natural‑language instructions, but existing tools are either tied to a single LLM vendor or require heavyweight UI‑automation frameworks. The gap is a lightweight, pluggable bridge that lets any LLM (local or remote) control the platform via ADB or OS‑level input.

What This Does

clickclickclick provides that bridge. The core CLI lives in main.py and orchestrates three layers:

  • Planner – selects a model (OpenAI, Gemini, Ollama) and turns a task prompt into a sequence of actions (clickclickclick/planner/*.py).
  • Finder – uses a vision model to locate UI elements in screenshots (clickclickclick/finder/*.py).
  • Executor – issues the actual input events (clickclickclick/executor/android.py for ADB, clickclickclick/executor/osx.py for macOS).

Configuration files under clickclickclick/config/ (models.yaml, function_declarations/*.yaml, prompts.yaml) define available actions and model credentials. Utility helpers (clickclickclick/utils.py) handle image encoding and resizing.

How It Is Wired

Execution begins at main.py:run (line 39). run parses CLI flags, loads the platform config via get_config (in clickclickclick/config/__init__.py), and builds a planner instance (Planner from clickclickclick/planner/__init__.py).

The planner calls llm_response → build_prompt → element_finder_prompt, then issues type_text, click_at_a_point, scroll, or screenshot.

  • type_text (in clickclickclick/executor/__init__.py) forwards to run_adb_command (Android) or the macOS equivalents; run_adb_command is the busiest leaf, called from 15 distinct places.
  • screenshot captures the screen, then resize (in clickclickclick/finder/__init__.py) prepares the image for the finder model.
  • find_element (in clickclickclick/finder/__init__.py) encodes the image (encode_image_to_base64) and invokes the selected finder (clickclickclick/finder/openai.py, gemini.py, etc.).

The most connected module is clickclickclick/config/__init__.py (imported by 13 other modules). Its instability is low (0.07), but changes here have a wide blast radius. No circular imports were detected.

Deep nesting (up to 11 levels) appears in clickclickclick/executor/osx.py, clickclickclick/finder/__init__.py, and clickclickclick/finder/openai.py, making the control flow hard to follow. The executor also opens files without a context manager (clickclickclick/executor/android.py). Several small code blocks are duplicated across api.py, main.py, clickclickclick/utils.py, and the executors, increasing maintenance cost.

How To Use It

# Clone the exact repo
git clone https://github.com/moses-y/clickclickclick
cd clickclickclick

# Recommended virtual environment
python3 -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Configure model keys in clickclickclick/config/models.yaml (e.g., OPENAI_API_KEY).

# Initialise the chosen planner/finder models
python main.py setup

Run a task:

python main.py run "Open Gmail and draft a message to rob@gmail.com" \
    --platform=android \
    --planner-model=gemini \
    --finder-model=gemini \
    --image-quality=45

The command triggers the flow described above, ending with ADB commands that move the cursor, type, or click.

Real‑World Use

A CI pipeline can invoke the CLI to automate UI regression checks on a device farm. Example snippet:

from clickclickclick.executor import AndroidExecutor

executor = AndroidExecutor()
executor.run_adb_command("adb install myapp.apk")
executor.type_text("Open Settings")
executor.click_at_a_point(300, 500)   # taps “Wi‑Fi”

This integrates the library into existing test harnesses without rewriting platform‑specific code.

Code Health & Issues

  • High – No lockfile – pyproject.toml has no accompanying lockfile; transitive dependencies are not pinned.
  • High – No CI – Repository lacks any GitHub Actions or other pipeline; changes are not automatically built or tested.
  • Medium – No Dependabot – No automated update bot for the two manifests.
  • Medium – Deep nesting – Files executor/osx.py, finder/__init__.py, finder/openai.py reach 11 indentation levels, hurting readability.
  • Medium – File handle leak – executor/android.py opens files without a with block.
  • Medium – Hub module – clickclickclick/config/__init__.py is imported by 13 modules; volatility here impacts many callers.
  • High – Duplicated code – Repeated 6‑line blocks across 12 files (e.g., API request wrappers) should be centralized.

No tests are missing; two test files exist (tests/test_android.py, tests/test_osx.py). License is present; no secrets were detected.

The Bottom Line

clickclickclick delivers a functional, model‑agnostic UI‑automation stack with a clear CLI entry point, but the codebase suffers from readability issues, duplicated logic, and missing production safeguards (lockfile, CI). It is suitable for teams comfortable with Python who need a quick, extensible bridge to LLM‑driven device control, provided they invest in refactoring the hot spots and add standard DevOps tooling.