The Problem

People who want hands‑free dictation on macOS need a local solution that can read lips, correct errors, and paste the result into any app without a microphone. Existing cloud services violate privacy and have latency; a local, GPU‑accelerated pipeline is hard to find.

What This Does

Lipflow captures the webcam feed, runs an on‑device visual speech recognizer (VSR) on Apple Silicon, optionally cleans the output with a local or remote LLM, and pastes the text at the cursor when a hotkey is released. The code is bundled as a native macOS app (/Applications/Lipflow.app) but can also run from the command line with uv run lipflow.

Key files:

  • lipflow/__main__.py – entry point that starts the Flask‑based UI.
  • lipflow/app.py – defines the Flask routes and launches the hotkey listener.
  • lipflow/av.py – grabs frames, extracts mouth crops and audio.
  • lipflow/vsr.py – runs the Apple‑GPU model from the espnet/ tree.
  • lipflow/cleanup.py – sends top‑3 VSR guesses to an LLM.
  • lipflow/paste.py – uses accessibility APIs to paste into the focused window.
  • lipflow/hotkey.py – monitors the modifier key that starts/stops recording.

How It Is Wired

  • Entry Point – uv run lipflow executes lipflow/__main__.py.
  • Main Flow – __main__ calls app.run(), which starts the Flask server and the background hotkey thread.
  • Hotkey → Capture – hotkey.py emits a START event; av.py begins a 25 fps capture loop.
  • Capture → VSR – Each frame is fed to vsr.py, which calls the espnet/nets/pytorch_backend/e2e_asr_transformer_av.py encoder on the GPU.
  • VSR → Cleanup – The raw top‑3 strings are passed to cleanup.py. When an LLM backend is available (Claude, Qwen3 via MLX, or Ollama), the best guess is chosen.
  • Cleanup → Paste – The finalized sentence is handed to paste.py, which uses the Accessibility API to insert the text at the current cursor position.
  • Side Effects –
  • corrections.py stores user‑fixed clips in ~/Library/Application Support/Lipflow/.
  • train_lm.py and train_vsr.py are invoked from the menu to fine‑tune the language and face models.
  • All models and logs live under ~/Library/Application Support/Lipflow/ and ~/Library/Logs/Lipflow.log.

The graph has a single hotkey → capture → VSR → cleanup → paste path; the only cycle is the hotkey listener, which is isolated.

How To Use It

Setup

git clone https://github.com/moses-y/lipflow.git ~/code/lipflow
cd ~/code/lipflow
./setup.sh      # installs uv deps, downloads 2 GB of models, builds Lipflow.app

Configuration No external config files are required; optional LLM keys are read from environment variables (ANTHROPIC_API_KEY, LIPFLOW_LOCAL_MODEL, etc.).

Running From the UI – open /Applications/Lipflow.app. From the terminal – uv run lipflow (the terminal window will request permissions).

Hotkey – Hold the right‑option key (⌥) while mouthing words; release to paste.

Real‑World Use

A writer in a voice‑poor environment can keep a Mac open and type silently. The pipeline runs entirely on the Mac’s GPU, so no network traffic is generated. The user can train the face model on a handful of practice sentences, improving accuracy for their own mouth shape.

# Example: force a re‑train from the command line
uv run lipflow --train-face

Code Health & Issues

  • No CI/CD – the repo has no .github/ workflow or other CI config; automated testing is absent.
  • Dependencies – pyproject.toml declares all packages but lacks a lockfile (uv.lock exists but is not pinned).
  • Tests – 6 test modules exist (tests/test_*) but cover only hotkey, paste, and cleanup logic; no functional tests for VSR or training.
  • License – MIT; no license file included in the repo root.

The Bottom Line

Lipflow offers a fully local, GPU‑accelerated lip‑reading workflow for macOS, with optional LLM cleanup. The code is modular but relies on external training scripts and a missing CI pipeline. It is suitable for users who need privacy‑preserving dictation and are comfortable with terminal setup.