The Problem

Automation of Android devices typically requires hand‑crafted UI scripts or brittle image‑matching pipelines. Teams that want to issue natural‑language commands need a way to translate visual context into reliable ADB actions without writing low‑level code.

What This Does

PhoneDriver is a Python prototype that closes the loop between a vision‑language model (Qwen‑3‑VL) and an Android device.

  • phone_agent.py implements the CLI driver (PhoneAgent) that orchestrates screenshot capture, model inference, and ADB command execution.
  • qwen_vl_agent.py wraps the Qwen‑3‑VL model, exposing analyze_screenshot, _generate_action, and _parse_action to turn a screen image into a concrete action (tap, swipe, type, wait).
  • ui.py launches a Gradio web UI (main at line 472) that forwards user‑entered natural‑language tasks to PhoneAgent.
  • Helper utilities (qwen_vl_utils.py) convert raw bytes to Pillow images and post‑process model output.

The repo ships a minimal config.json for screen dimensions, model temperature, and step delay, and a README.md that documents the required environment (Python 3.10+, ADB, GPU).

How It Is Wired

Execution starts in ui.py → main() (line 472). The UI collects a task string, builds a PhoneAgent instance, and calls PhoneAgent.execute_task(task).

PhoneAgent.__init__ runs four setup helpers (_setup_logging, _setup_directories, _check_adb_connection, _verify_screen_resolution) and instantiates QwenVLAgent.

execute_task enters a loop that:

  1. Calls capture_screenshot → adb exec-out screencap -p → _run_adb_command (the most‑called internal function, invoked from six places).
  2. Passes the image to QwenVLAgent.analyze_screenshot.
  3. Receives a textual action, which QwenVLAgent._generate_action parses via _parse_action.
  4. Dispatches to one of the concrete executors (_execute_tap, _execute_swipe, _execute_type, _execute_wait), each of which ultimately calls _run_adb_command to issue the ADB instruction.

The internal call graph records 39 intra‑repo edges; the hub functions are _run_adb_command (6 callers) and capture_screenshot (3 callers). No circular imports exist, and the module dependency graph is shallow: ui → phone_agent → qwen_vl_agent → qwen_vl_utils.

Deep nesting (max indentation depth 8) appears in ui.py, phone_agent.py, and qwen_vl_agent.py, making the control flow harder to follow. The most volatile module is phone_agent.py, which both imports and is imported by a single other module, giving it an instability score of 0.5.

How To Use It

# Clone the fork that contains the current code
git clone https://github.com/moses-y/PhoneDriver
cd PhoneDriver

# Create and activate a virtual environment
python -m venv phonedriver
source phonedriver/bin/activate

# Install the required Python packages
pip install git+https://github.com/huggingface/transformers
pip install pillow gradio qwen_vl_utils requests
  • Verify ADB can see the device: adb devices.
  • Adjust config.json if the automatic resolution detection is insufficient.
  • Web UI: python ui.py → open http://localhost:7860 and type a task.
  • CLI: python phone_agent.py "Open the camera app".

No Dockerfile, CI configuration, or test suite is present, so the above steps are the only documented path to a running instance.

Real‑World Use

A QA team could embed PhoneDriver in a larger test harness: the harness calls PhoneAgent.execute_task for each high‑level test step, captures the resulting logs, and asserts that the final screen matches an expected visual fingerprint. Because actions are generated by a language model, the same test can be expressed in plain English without maintaining a separate script for each UI version.

Code Health & Issues

  • High – cognitive_load – Deep nesting (max indent 8) in ui.py, phone_agent.py, qwen_vl_agent.py. Refactor with early returns or helper functions.
  • Medium – resource_safety – Files opened without a context manager in qwen_vl_agent.py and qwen_vl_utils.py. Replace open(...) with with open(...) as f:.
  • Medium – resilience – Broad except: clauses in qwen_vl_agent.py swallow unexpected errors. Scope catches to specific exception types and log/re‑raise others.
  • Medium – SDLC – No test files detected; code paths are unvalidated by automated tests.
  • Medium – SDLC – No CI/CD pipeline; no automated linting or build gate.

No secrets were found, and a LICENSE file is present. The repo lacks a requirements.txt or pyproject.toml; dependency versions are only hinted at in the README.

The Bottom Line

PhoneDriver demonstrates a functional end‑to‑end loop from natural‑language intent to ADB actions, with a clear separation between UI, agent orchestration, and model inference. The codebase is small but suffers from deep nesting, unsafe file handling, and a lack of tests or CI. It is suitable for experimental or proof‑of‑concept integration, but production use will require refactoring, proper testing, and automated validation pipelines.