The Problem

Meta Ray-Ban smart glasses lack an integrated, real-time transcription and AI assistant pipeline that works out of the box. Developers building on this platform must stitch together speech-to-text, speaker diarization, vision+voice AI, and WebRTC streaming themselves, with no reference implementation that addresses authentication, build reproducibility, or deployment safety.

What This Does

GlassFlow is a 156-file reference implementation for real-time transcription and AI assistance on Meta Ray-Ban glasses. The iOS component lives in samples/CameraAccess/CameraAccess/ (Swift, 43 files), Android in samples/CameraAccessAndroid/app/ (Kotlin, 33 files). It streams PCM audio (16kHz mono) from glasses or phone mic to Deepgram Nova-3 via WebSocket for live transcription with speaker diarization (S1, S2...). Optional Gemini Live provides voice+vision AI. WebRTC streaming shares the glasses POV to a browser viewer via samples/CameraAccess/server/index.js. Entry points are samples/CameraAccess/server/index.js (Node/Express) and the iOS CameraAccess.xcodeproj or Android Studio project. The code touches Deepgram’s streaming API, Gemini Live runtime, and WebRTC peer connections; no lockfile exists beside samples/CameraAccess/server/package.json, and a committed private key (samples/CameraAccessAndroid/app/sample.keystore) poses a critical rotation risk.

How It Is Wired

Audio flows from samples/CameraAccess/CameraAccess/Transcription/DeepgramService.swift (iOS) or samples/CameraAccessAndroid/app/src/main/java/.../gemini/AudioManager.kt (Android) into Deepgram’s WebSocket streaming API. Partial transcripts update the live UI; final transcripts render in the scrolling panel with speaker labels. The server at samples/CameraAccess/server/index.js accepts WebRTC connections and serves public/index.html for browser viewing. Gemini integration lives in samples/CameraAccess/CameraAccess/Gemini/GeminiLiveService.swift (iOS) and samples/CameraAccessAndroid/app/src/main/java/.../gemini/GeminiLiveService.kt (Android), both flagged for high cognitive load (deep nesting x11, 91 branch points over 297 lines). The module graph has 77 analyzed files with 0 import edges and 0 circular dependencies, but the most connected module samples/CameraAccess/server/index has instability 0 (no imports, no dependents). Duplicated 6-line blocks appear 17 times across 10 files, including HomeScreenView.swift and NonStreamView.swift.

How To Use It

Setup: Clone with git clone https://github.com/moses-y/GlassFlow.git. Open samples/CameraAccess/CameraAccess.xcodeproj for iOS or samples/CameraAccessAndroid/ in Android Studio.

Configuration: Copy example secrets files. For iOS: cp samples/CameraAccess/CameraAccess/Secrets.swift.example samples/CameraAccess/CameraAccess/Secrets.swift and edit with Deepgram and optional Gemini API keys. For Android: cp samples/CameraAccessAndroid/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/Secrets.kt.example .../Secrets.kt and add github_token to samples/CameraAccessAndroid/local.properties for the Meta DAT SDK via GitHub Packages.

Running it: iOS—select your iPhone target and run (Cmd+R). Android—let Gradle sync, select your phone, run (Shift+F10). Without glasses, tap "Start on iPhone" or "Start on Phone" and use the phone mic; with glasses, enable Developer Mode in the Meta AI app first.

Secrets: The committed sample.keystore at samples/CameraAccessAndroid/app/sample.keystore must be rotated and purged from history immediately.

Real-World Use

A developer iterates on speaker diarization accuracy by running the app with two phones side-by-side, tapping Scribe to stream live transcripts to a browser viewer via WebRTC. The transcript scrolls in real-time with S1/S2 labels, and they toggle Gemini Live to test vision+voice queries on the glasses POV. The server Dockerfile at samples/CameraAccess/server/Dockerfile builds the streaming endpoint; the team can validate changes by pushing to a branch with CI (currently absent).

Code Health & Issues

  • CRITICAL - Rotate and purge the committed private key at samples/CameraAccessAndroid/app/sample.keystore; this is a test fixture that must not be in version control.
  • HIGH - Commit a lockfile beside samples/CameraAccess/server/package.json; without it, transitive dependency versions can diverge between development and production.
  • HIGH - Add a CI workflow that builds and tests this repository; 77 source files with no automated gate means every merge runs unvalidated.
  • HIGH - Add a build gate for deployable artifacts at samples/CameraAccess/server/Dockerfile; deployment images have never been validated by an automated step.
  • MEDIUM - Pin the container base image by digest in samples/CameraAccess/server/Dockerfile; node:20-alpine is mutable, so builds contain different libc and CVEs over time.
  • MEDIUM - Expand the test suite; 1 test file against 80 source files gives a coverage ratio of 0.013, far too low to trust a green badge.
  • MEDIUM - Add a pre-commit secret gate; a secret-shaped file is tracked with no repo-level scanning, so the same class of leak will recur.
  • LOW - Add convention files (.editorconfig, .gitattributes, formatter config) to prevent editor-level inconsistency across contributors.

The Bottom Line

This repo provides a functional, well-structured reference implementation for real-time transcription and AI on Ray-Ban glasses, with clear audio pipelines, WebRTC streaming, and optional Gemini vision+voice. However, it ships with a committed private key, no lockfile, no CI/CD, and a Dockerfile running as root from a mutable base image—issues that must be resolved before it reaches production. It is suitable for developers who need a starting point for glass-connected transcription and who can address the hygiene gaps independently.