The Problem
Recognizing hand gestures and facial expressions from a webcam in real time on an Apple Silicon Mac requires a fast, token‑free model that can run locally. Existing solutions either off‑load the work to the cloud or generate conversational tokens, adding latency and privacy risk.
What This Does
The repository delivers a zero‑token, on‑device decision engine that processes each webcam frame with Liquid AI’s d1‑2B model.
index.htmlserves a lightweight web UI that captures frames, posts them to/run, and visualises the four question outputs.decide.pyis the server: it receives the JPEG, forwards it to the chosen backend (MLX, Core AI GPU, or Core AI Neural Engine), and streams back logits and bounding boxes.- Core AI paths (
coreai/swift/…) contain a small Swift worker that loads the Core AI‑exported model and exposes aPOST /run?backend=coreai‑...endpoint. coreai/coreai_backend.pyorchestrates backend selection, model loading, and request handling.- The model weights are downloaded from Hugging Face on first run; no frame data leaves the host.
How It Is Wired
Entry point: index.html → JavaScript → POST /run. Server side (decide.py):
decide.pyparses the request and callscoreai_backend.run(request, backend).coreai_backend.runloads the appropriate engine viacoreai_backend.load_backend(backend); if the Swift worker is not yet built, it falls back to MLX.- The loaded backend executes
model.system_one(frame, QUESTIONS); for MLX this is a single forward pass inmlx-vlm; for Core AI it calls the Swift worker’s exported function. - Results (logits + boxes) are returned to
decide.py, which formats JSON for the browser.
File responsibilities
coreai/check_torch.pyverifies PyTorch compatibility before exporting.coreai/export_gpu.pyconverts weights for Core AI GPU.coreai/export_ane.pydoes the same for the Neural Engine.coreai/d1_torch.pyholds the PyTorch wrapper used only during export.
The Swift worker (coreai/swift/Sources/d1worker/main.swift) receives the 640×360 frame, runs the Core AI model, and returns logits and boxes.
How To Use It
Setup
# Clone exactly as provided
git clone https://github.com/moses-y/mlx-d1-gestures
cd mlx-d1-gestures
# Install local dependencies
./start.sh
start.sh runs uv run decide.py and opens http://127.0.0.1:8049 once the model is ready. Configuration
DECIDE_PORT(default 8049) andDECIDE_MODELcan be set indecide.py.- To use Core AI, build the Swift worker:
cd coreai/swift && swift build -c release
Running The web UI is the single entry point. Press Start camera, grant permission, and the page will stream webcam frames to the backend.
Real‑World Use
A retail kiosk can deploy this on a Mac mini: the kiosk’s camera feeds the UI; the app classifies “thumbs up”, “peace”, or a smiling face to trigger different promotional content. No external API calls, preserving privacy and responsiveness.
Code Health & Issues
- Medium – SDLC – No test files (
tests/missing); Severity: Med –^ - Medium – SDLC – No CI/CD configuration (
.github/absent); Severity: Med –^ - Medium – SDLC – Missing LICENSE file; Severity: Med –
^
No other static issues detected. The repository contains 21 files, 13 Python, 2 Swift, and a single Swift worker, all under 1 kB each.
The Bottom Line
The project demonstrates a practical, fully local gesture‑recognition pipeline on Apple Silicon. It’s lightweight, privacy‑preserving, and easy to run with a single shell script. The lack of tests, CI, and a license limits its readiness for production use, but the core logic is clear and well‑contained. Engineers who need deterministic, token‑free inference on macOS will find it useful, provided they add their own validation and packaging steps.