The Problem

Recognizing hand gestures and facial expressions from a webcam in real time on an Apple Silicon Mac requires a fast, token‑free model that can run locally. Existing solutions either off‑load the work to the cloud or generate conversational tokens, adding latency and privacy risk.

What This Does

The repository delivers a zero‑token, on‑device decision engine that processes each webcam frame with Liquid AI’s d1‑2B model.

  • index.html serves a lightweight web UI that captures frames, posts them to /run, and visualises the four question outputs.
  • decide.py is the server: it receives the JPEG, forwards it to the chosen backend (MLX, Core AI GPU, or Core AI Neural Engine), and streams back logits and bounding boxes.
  • Core AI paths (coreai/swift/…) contain a small Swift worker that loads the Core AI‑exported model and exposes a POST /run?backend=coreai‑... endpoint.
  • coreai/coreai_backend.py orchestrates backend selection, model loading, and request handling.
  • The model weights are downloaded from Hugging Face on first run; no frame data leaves the host.

How It Is Wired

Entry point: index.html → JavaScript → POST /run. Server side (decide.py):

  1. decide.py parses the request and calls coreai_backend.run(request, backend).
  2. coreai_backend.run loads the appropriate engine via coreai_backend.load_backend(backend); if the Swift worker is not yet built, it falls back to MLX.
  3. The loaded backend executes model.system_one(frame, QUESTIONS); for MLX this is a single forward pass in mlx-vlm; for Core AI it calls the Swift worker’s exported function.
  4. Results (logits + boxes) are returned to decide.py, which formats JSON for the browser.

File responsibilities

  • coreai/check_torch.py verifies PyTorch compatibility before exporting.
  • coreai/export_gpu.py converts weights for Core AI GPU.
  • coreai/export_ane.py does the same for the Neural Engine.
  • coreai/d1_torch.py holds the PyTorch wrapper used only during export.

The Swift worker (coreai/swift/Sources/d1worker/main.swift) receives the 640×360 frame, runs the Core AI model, and returns logits and boxes.

How To Use It

Setup

# Clone exactly as provided
git clone https://github.com/moses-y/mlx-d1-gestures
cd mlx-d1-gestures

# Install local dependencies
./start.sh

start.sh runs uv run decide.py and opens http://127.0.0.1:8049 once the model is ready. Configuration

  • DECIDE_PORT (default 8049) and DECIDE_MODEL can be set in decide.py.
  • To use Core AI, build the Swift worker:
cd coreai/swift && swift build -c release

Running The web UI is the single entry point. Press Start camera, grant permission, and the page will stream webcam frames to the backend.

Real‑World Use

A retail kiosk can deploy this on a Mac mini: the kiosk’s camera feeds the UI; the app classifies “thumbs up”, “peace”, or a smiling face to trigger different promotional content. No external API calls, preserving privacy and responsiveness.

Code Health & Issues

  • Medium – SDLC – No test files (tests/ missing); Severity: Med – ^
  • Medium – SDLC – No CI/CD configuration (.github/ absent); Severity: Med – ^
  • Medium – SDLC – Missing LICENSE file; Severity: Med – ^

No other static issues detected. The repository contains 21 files, 13 Python, 2 Swift, and a single Swift worker, all under 1 kB each.

The Bottom Line

The project demonstrates a practical, fully local gesture‑recognition pipeline on Apple Silicon. It’s lightweight, privacy‑preserving, and easy to run with a single shell script. The lack of tests, CI, and a license limits its readiness for production use, but the core logic is clear and well‑contained. Engineers who need deterministic, token‑free inference on macOS will find it useful, provided they add their own validation and packaging steps.