livecc

The Problem

The repository is a collection of five loosely‑coupled projects with no automated test gate, no CI/CD pipeline, and no lockfile for its dependencies. The codebase also lacks a licence file, so redistribution rights are undefined, and several files open handles without context managers or swallow exceptions broadly. These gaps mean any change can introduce regressions that go undetected, and the code cannot be legally reused in client engagements without a clear licence.

What This Does

LiveCC is a portfolio of five self‑contained projects (evaluation, webpage, demo, livecc‑utils, utils) that collectively deliver a video‑LLM capable of real‑time commentary via a streaming speech‑transcription pipeline. The core technical contribution is a novel video‑ASR streaming method that achieves SOTA on both streaming and offline benchmarks. The largest, most‑used sub‑projects are:

• evaluation/ – 25 files, 13 code files; runs VQA, MCQ, and caption generation on benchmarks such as LiveSports‑3K, MVBench, and OVObatch.
• demo/ – 8 files, 5 code files; provides a Gradio web UI (demo/app.py) and a CLI (demo/cli.py) for live inference.
• livecc‑utils/ – 8 files, 3 code files; defines pyproject.toml and shared utilities (e.g., utils/multiprocessor.py which is imported by 13 other modules).
• data/production/ – contains the data‑generation pipeline (distributed_lighter_asd/, distributed_lmm4asd.py, etc.) that builds the 526K‑utterance WhisperX dataset and processes video‑LLM inputs.
• webpage/ – 17 files, 6 code files; the static site and JS assets (notably webpage/static/js/bulma-carousel.js at 1940 lines) that host the Hugging Face demo and paper metadata.

Technique recurring across the projects: multiprocessor worker patterns, streaming conversation preprocessing (get_phrase_before_timestamp, load_conversation), and a shared model backbone (data/production/distributed_lighter_asd/model/Model.py) that fuses visual and audio front‑ends.

How It Is Wired

Execution starts at the two entry points documented in the README:

The internal call graph (85 resolved edges) shows a hub at utils/multiprocessor.py – 13 modules import it, giving it a high‑blast‑radius. Other frequently called functions are local_mt (6 callers), _spatial_resize_video (4), parse_args (3), and LiveCCDemoInfer (3). Paths from an entry point to external side‑effects are short: main → os.makedirs for filesystem writes; the demo routes through Gradio callbacks that are invisible to the static graph.

Key files and their responsibilities (from the “WHAT EACH FILE IS RESPONSIBLE FOR” block):

• data/lmm_dataset.py – defines LMMDataset, readlastline, bytes_to_pil, get_phrase_before_timestamp; 13 functions, reads/writes files, calls 9 internal helpers.
• demo/infer.py – __init__, live_cc, video_qa; calls a model for inference, imported by demo/app.py.
• utils/multiprocessor.py – 13 importers; a small but heavily‑used worker pool.
• data/production/distributed_lighter_asd/main.py – argument parsing, orchestration of the data‑production pipeline.

How To Use It

Clone (verbatim):

git clone https://github.com/moses-y/livecc.git
cd livecc

Installation (from README, Python >= 3.11 required):

pip install torch torchvision torchaudio
pip install "transformers>=4.52.4" accelerate deepspeed peft opencv-python decord datasets tensorboard gradio pillow-heif gpustat timm sentencepiece openai av==12.0.0 qwen_vl_utils liger_kernel numpy==1.24.4
pip install flash-attn --no-build-isolation
pip install livecc-utils==0.0.2

Advanced data‑production dependencies (add if you want to rebuild the dataset):

pip install insightface onnxruntime-gpu python_speech_features wavfile

Gradio demo (launches a streaming video‑LLM UI):

python demo/app.py --js_monitor
# –js_monitor uses JavaScript timestamp monitoring; disable for high‑latency environments

CLI (bare‑bones command‑line inference):

python demo/cli.py

Hands‑on inference – see the repo’s inference.md for detailed script usage.

Training – single‑node scripts under scripts/pt_local.sh; multi‑node setups use torchrun and --gradient_accumulation_steps.

Configuration – environment variables documented in the README (e.g., VIDEO_MIN_PIXELS, FPS_MAX_FRAMES, VIDEO_MAX_PIXELS). No separate config file is required; the variables are read at runtime from the entry‑point scripts.

Real‑World Use

A production team that needs a real‑time video‑commentary service can spin up the Gradio demo (demo/app.py) behind a reverse proxy, feed it a live video source, and stream transcribed speech to the LLM. The data‑production pipeline (data/production/distributed_lighter_asd/) can be re‑run to refresh the WhisperX‑526K dataset when new audio‑video corpora become available, keeping the model’s ASR front‑end up‑to‑date without touching the LLM weights.

Code Health & Issues

Measured static‑analysis findings (deterministic, from the pipeline):

• [HIGH/clarity] Duplicated code blocks – 157 repeated 6‑line fragments across 15 files (data/lmm_dataset.py, data/production/distributed_lmm4asd.py, data/production/distributed_lighter_asd/multiprocessor.py, utils/multiprocessor.py). Extract shared helpers to DRY the logic.
• [HIGH/cognitive_load] Deep nesting – max indentation depth 7 in data/lmm_dataset.py, data/production/distributed_lighter_asd/face_tracker.py, data/production/distributed_lighter_asd/main.py. Flatten with guard clauses / early returns.
• [HIGH/cognitive_load] Oversized file – webpage/static/js/bulma-carousel.js at 1940 lines; split into cohesive modules by responsibility.
• [MEDIUM/resource_safety] File opened without context manager x4 – evaluation/distributed_mcq_predictor.py, data/production/language_detect.py, demo/cli.py. Wrap open(...) in with open(...) as f:.
• [MEDIUM/resilience] Broad exception handling x4 – evaluation/distributed_mcq_predictor.py, evaluation/mvbench/check_video_exists.py, evaluation/mvbench/distributed_evaluate_mvbench.py. Catch specific exceptions; log or re‑raise the rest.
• [MEDIUM/resilience] Empty catch block – webpage/static/js/bulma-carousel.js. At least log the error; never silently discard.
• [MEDIUM/cognitive_load] High branching density x3 – data/production/distributed_lighter_asd/asd_utils.py, debug.py, webpage/static/js/bulma-slider.js (15 branch points over 42 lines). Decompose decision logic; consider a dispatch table.

SDLC hygiene (detected, do not contradict):

• No test files – untested code paths repository‑wide.
• No CI/CD pipeline – no automated build/test gate.
• No lockfile beside livecc-utils/pyproject.toml – builds are non‑reproducible.
• No licence file at the repository root – redistribution rights undefined.
• No Dependabot/Renovate configuration – dependencies can go unpatched.

The Bottom Line

LiveCC is a focused portfolio of five projects that deliver a state‑of‑the‑art video‑LLM with streaming ASR. The codebase is functional and well‑documented for its intended use‑cases (Gradio demo, CLI, data‑production pipelines), but it ships without a licence, test suite, CI gate, or lockfile, and several files contain resource‑safety and resilience antipatterns. It is suitable for research prototyping and rapid demos, but anyone planning to integrate or extend the code in a production system should first add a licence, commit a lockfile, introduce a minimal test harness, and refactor the duplicated / deeply‑nested blocks highlighted above. Teams that can absorb those investments will gain a reusable, well‑scoped video‑LLM stack; those needing out‑of‑the‑box production readiness may need to supplement with additional engineering effort.