The Problem

VisionClaw is a prototype real‑time AI assistant for Meta Ray‑Ban smart glasses that couples voice, vision and agentic actions via Gemini Live and OpenClaw. The codebase has no automated test or CI gate, duplicated logic in view files, and several decision‑heavy Swift files with high branching density, making future changes risky without a safety net.

What This Does

VisionClaw lets a user tap an AI button and ask “What am I looking at?” – Gemini’s Live API streams video frames and audio, returns a description, and can invoke OpenClaw‑backed actions such as adding items to a shopping list or sending a message. The iPhone‑mode branch uses the device’s back camera instead of the glasses.

Key files and their responsibilities:

  • samples/CameraAccess/CameraAccess/Gemini/GeminiLiveService.swift – orchestrates the WebSocket connection to Gemini, forwards JPEG frames (~1 fps) and PCM audio, and handles tool‑call execution.
  • samples/CameraAccess/CameraAccess/Gemini/GeminiConfig.swift – stores the Gemini API key; the placeholder static let apiKey = "YOUR_GEMINI_API_KEY" must be replaced.
  • samples/CameraAccess/CameraAccess/OpenClaw/OpenClawBridge.swift – thin bridge that forwards Gemini tool calls to the OpenClaw gateway (runs locally or on a LAN‑bound host).
  • samples/CameraAccess/CameraAccess/OpenClaw/ToolCallModels.swift – defines the Swift structs that map Gemini’s tool‑call payloads; contains 98 branch points over 321 lines, indicating dense conditional logic.
  • samples/CameraAccess/CameraAccess/OpenClaw/ToolCallRouter.swift – dispatches a parsed tool call to the appropriate OpenClaw skill (e.g., web search, messaging).
  • View models – StreamSessionViewModel.swift, WearablesViewModel.swift, DebugMenuViewModel.swift manage UI state and session lifecycle.
  • Views – HomeScreenView.swift and NonStreamView.swift each contain a repeated 6‑line block (duplicate code).
  • samples/CameraAccess/CameraAccess/Gemini/AudioManager.swift – manages the bidirectional audio pipeline (16 kHz capture, 24 kHz playback).

Outside the repo, Gemini Live contacts Google’s API; OpenClaw contacts a local gateway that may call web services, messaging apps, or smart‑home controllers.

How It Is Wired

Execution starts in CameraAccessApp.swift (the Xcode entry point). Tapping the AI button triggers GeminiLiveService.startSession(), which opens a WebSocket to Gemini. Video frames are captured by IPhoneCameraManager.swift (or the glasses DAT SDK), encoded as JPEGs, and sent alongside audio from AudioManager.swift. Gemini replies with either an audio PCM stream (played through the iOS speaker) or a tool‑call payload.

The tool‑call payload flows:

  1. GeminiLiveService.swift parses the JSON → ToolCallModels.swift structs.
  2. ToolCallRouter.swift matches the action name and calls OpenClawBridge.swift.
  3. OpenClawBridge.swift POSTs to the gateway (http://<host>:18789/…) and returns a text response.
  4. The response is fed back into the Gemini conversation, and Gemini speaks the result.

A hub exists at GeminiLiveService.swift: it simultaneously manages network I/O, audio buffering, and tool‑call routing, giving it a wide blast radius. The high branching density (98 branches over 321 lines) in ToolCallModels.swift similarly means many conditional paths can be triggered by tool names or payload shapes.

How To Use It

Clone & open

git clone https://github.com/moses-y/VisionClaw.git
cd VisionClaw/samples/CameraAccess
open CameraAccess.xcodeproj

Add Gemini key – edit samples/CameraAccess/CameraAccess/Gemini/GeminiConfig.swift and replace the placeholder with your key from Google AI Studio.

Build & run – select an iPhone target and press Cmd+R.

iPhone mode – tap “Start on iPhone” to use the device camera; then tap the AI button to start a Gemini Live session.

With glasses – pair the Ray‑Ban glasses via the Meta AI app (enable Developer Mode), tap “Start Streaming”, then the AI button.

Optional OpenClaw – follow the OpenClaw setup guide; ensure ~/.openclaw/openclaw.json contains a gateway with bind: "lan" and a token, enabling the chatCompletions endpoint. Once running, the iPhone can reach the gateway on the local network and Gemini will be able to invoke the 56+ skills (web search, messaging, smart‑home, etc.).

Real‑World Use

A user puts on the glasses, taps the AI button, and says, “Add milk to my shopping list.” GeminiLiveService.swift forwards the request to the OpenClaw gateway, which contacts the Notes app on the phone and adds “milk”. The AI then confirms, “Milk added to your list.” If the user instead asks “What’s the weather forecast for tomorrow?” Gemini’s visual description of the scene is combined with a web‑search skill via OpenClaw, and the spoken answer references the current location.

Code Health & Issues

  • Duplicated code – 6‑line blocks appear in samples/CameraAccess/CameraAccess/Views/HomeScreenView.swift and samples/CameraAccess/CameraAccess/Views/NonStreamView.swift. Extract a shared helper to DRY the logic.
  • High branching density – samples/CameraAccess/CameraAccess/Gemini/GeminiLiveService.swift and samples/CameraAccess/CameraAccess/OpenClaw/ToolCallModels.swift each contain ~98 branch points over ~321 lines; decompose decision logic into a strategy or table‑driven dispatch.
  • Missing test suite – 32 source files, zero test files. Any change ships without signal that existing behaviour is preserved.
  • Missing CI workflow – no .github/ or CI configuration; every merge runs untested.
  • Repo hygiene – licence present; no Dockerfile, no lockfile, no committed secrets.

The Bottom Line

VisionClaw is a functional prototype that demonstrates voice‑plus‑vision AI on smart glasses and OpenClaw‑enabled actions. It lacks automated tests, CI, and has duplicated/over‑complex logic that raises maintenance risk. It is suitable for experimentation and quick demos, but should not be merged into a production codebase until a test suite and CI pipeline are added and the duplicated/high‑branching code is refactored.