The Problem
Real‑time video commentary requires a language model that can ingest a live video stream, transcribe speech on‑the‑fly, and generate coherent captions without buffering. Existing video‑LLM pipelines are batch‑oriented, incur high latency, and cannot scale to the data volumes needed for robust streaming performance.
What This Does
LiveCC implements a streaming video‑LLM that couples a visual encoder with a continuous ASR front‑end. The core training and inference code lives in the data/production/distributedlighterasd/ package:
facedetector.py / facetracker.py – extract and track visual tokens per frame. lightasd.py – orchestrates the visual‑ASR pipeline, feeding token streams to the LLM. Model definitions (model/Encoder.py, model/Classifier.py, model/Model.py) provide the multimodal backbone used in both pre‑training (scripts/ptlocal.sh) and instruction‑tuning (train.py).
A lightweight Flask‑based demo (demo/app.py) and a CLI wrapper (demo/cli.py) expose the model for interactive use. The repository also bundles evaluation scripts for benchmarks such as LiveSports‑3K and MVBench under evaluation/.
How To Use It
Setup
Python ≥3.11 pip install torch torchvision torchaudio pip install "transformers>=4.52.4" accelerate deepspeed peft \ opencv-python decord datasets tensorboard gradio \ pillow-heif gpustat timm sentencepiece openai av==12.0.0 \ qwenvlutils ligerkernel numpy==1.24.4 pip install flash-attn --no-build-isolation pip install livecc-utils==0.0.2 Optional for data pipeline pip install insightface onnxruntime-gpu pythonspeechfeatures wavfile
The repository was developed with torch==2.6.0, transformers==4.50.0, and liger-kernel==0.5.5; other compatible versions should work.
Configuration
Environment variables control the video token budget (see scripts/ptlocal.sh for examples):
VIDEOMINPIXELS – minimum visual tokens per frame (default 78400). FPSMAXFRAMES – max frames per video (default 480). VIDEOMAXPIXELS – upper bound on total visual tokens (default 19267584).
These can be exported in a shell before launching training or inference.
Running it
Demo (Gradio UI)
python demo/app.py --jsmonitor # use JS timestamp monitor; omit for high‑latency hosts
The UI streams a video, runs lightasd.py internally, and displays live captions.
CLI inference
python demo/cli.py
Accepts a video path and prints the generated commentary to stdout.
Pre‑training (single‑node)
bash scripts/ptlocal.sh
The script sets the environment variables above and launches torchrun with the arguments defined in train.py.
Distributed inference (benchmark)
Example entry point: data/production/distributedlighterasd/main.py – used by evaluation scripts such as evaluation/livesports3kcc/distributedgeneratelivecc.py.
Real‑World Use
A broadcast workflow could pipe live video frames into lightasd.py via a GStreamer pipeline, letting demo/app.py (or a custom Flask endpoint) return timestamps and captions for overlay graphics. The modular token budget parameters enable scaling from low‑bandwidth streams to 4 K footage without changing model code.
Code Health & Issues
Med – Untested code paths – No tests/ directory; core modules (lightasd.py, model files) lack unit tests. Med – Missing CI/CD – Repository contains no .github/workflows/ or other pipeline definitions. Low – No lockfile – Dependencies declared only in livecc-utils/pyproject.toml; reproducible builds rely on version pins in the README, not a lockfile. Low – License scope – Only a LICENSE file inside distributedlighter_asd/; top‑level project lacks an explicit license header. Low – Potential runtime errors – demo/app.py assumes a GPU‑available environment; no fallback or clear error messages if CUDA is missing.
Overall, the codebase is coherent and the entry points are well‑documented, but the lack of automated testing and CI means integration risk is higher for production deployments.
The Bottom Line
LiveCC delivers a functional streaming video‑LLM with clear demo and training scripts, suitable for research teams or enterprises that can allocate resources for validation and CI integration. The absence of tests, CI, and a top‑level license suggests caution before adopting it in critical pipelines.