The Problem

Real‑time facial landmark tracking on a CPU is still a bottleneck for many avatar‑driven applications that cannot rely on GPU resources. Existing solutions either require heavy dependencies or expose the camera feed to the host process, limiting deployment on low‑spec machines or in privacy‑sensitive environments.

What This Does

OpenSeeFace provides a CPU‑only pipeline that runs a MobileNetV3‑based ONNX model at 30‑60 fps. The core tracker lives in facetracker.py and tracker.py, while input_reader.py abstracts webcam or video file input. Unity integration is supplied in the Unity/ folder; the OpenSee.cs component receives UDP packets from the Python process and exposes the trackingData struct to Unity scripts such as OpenSeeShowPoints.cs.

Key files:

  • facetracker.py – CLI entry point, parses arguments, starts tracker.run.
  • tracker.py – orchestrates model loading (model.py), frame acquisition (input_reader.py), and post‑processing (similaritytransform.py).
  • model.py – loads ONNX files from models/ via onnxruntime.
  • Unity/OpenSee.cs – UDP listener, deserialises packet into OpenSeeTrackingData.

How It Is Wired

  1. Entry point – facetracker.py is executed (python facetracker.py …). It creates a Tracker object (tracker.Tracker) and calls run().
  2. Input – run() invokes input_reader.read_frame(). Depending on the command line, this calls either the DirectShow wrapper (dshowcapture.py) or the ESCAPI wrapper (escapi.py). Both are thin Python bindings to the native DLLs in dshowcapture/ and escapi/.
  3. Model inference – tracker.run loads the ONNX model via model.load() (calls onnxruntime.InferenceSession). The selected model file lives in models/ (e.g., lm_model4_opt.onnx).
  4. Post‑processing – Raw landmark tensors are transformed by similaritytransform.transform() and refined by retinaface.py (face detection) and remedian.py (noise reduction). These modules have no inbound imports, so their blast radius is limited to the tracker pipeline.
  5. Output – Processed landmarks are packed into a UDP packet (tracker.send_udp) and transmitted to the Unity side. The Unity listener in OpenSee.cs deserialises the packet and updates the public trackingData field.

The internal import graph shows a clear star topology: tracker imports three downstream modules (similaritytransform, retinaface, remedian) and is the only module with inbound imports (facetracker, input_reader). No circular dependencies exist, so a change in any downstream module affects only the tracker’s post‑processing stage.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/OpenSeeFace
cd OpenSeeFace

# Install Python dependencies (poetry lock present)
poetry install   # creates a virtualenv and installs onnxruntime, opencv, etc.

# Run the tracker (example uses default webcam)
poetry run python facetracker.py --model models/lm_model4_opt.onnx --output udp

# In Unity, open the sample scene under Unity/ and attach OpenSee.cs to a GameObject.
# The component will start listening on the default UDP port and drive the avatar.

If a pre‑built binary is preferred, the release assets contain facetracker.exe (Windows) that runs without a Python interpreter. No environment variables are required; all configuration is passed via command‑line flags (as defined in facetracker.py).

Real‑World Use

A VTuber studio runs the tracker on a dedicated low‑cost laptop (Intel i5, no GPU). The laptop streams UDP packets over the local network to a high‑end PC that hosts Unity. The Unity client receives the data via OpenSee.cs, driving a VRM avatar in real time while the camera feed never leaves the laptop, satisfying privacy constraints.

Code Health & Issues

  • High – Add a test suite – no test files despite 23 source files.
  • Medium – Least‑privilege GITHUB_TOKEN – workflow .github/workflows/create-binaries.yaml declares no permissions.
  • Medium – Enable Dependabot/Renovate – manifest present, no bot configured.
  • Medium – Dependency vulnerability gate – CI lacks a vulnerability scan step.
  • Medium – Large binaries in repo – ONNX models (*.onnx) exceed 5 MiB each, inflating clone size.
  • Low – Job timeout‑minutes missing – same workflow lacks explicit timeouts.
  • Low – Missing repo convention files – no .editorconfig, .gitattributes, or formatter config.

Additional observations: deep nesting (max depth 8) and duplicated 6‑line blocks across several C# and C++ files increase maintenance effort. Exception handling is overly broad in input_reader.py, retinaface.py, and tracker.py, which can mask runtime errors.

The Bottom Line

OpenSeeFace delivers a functional, CPU‑only facial landmark pipeline with a ready‑made Unity bridge, making it practical for low‑spec or privacy‑focused deployments. The codebase is usable but suffers from maintainability issues (deep nesting, duplicated blocks) and lacks automated testing and CI hardening. Engineers comfortable with Python, C++, and Unity can adopt it quickly, but should prioritize adding tests and tightening CI before extensive production use.