The Problem
Real-time video AI agents are hard to build because they require coordinating video ingestion, object detection, speech-to-text, LLM reasoning, and text-to-speech with strict latency budgets. Most existing frameworks assume a single edge provider or lock you into one LLM vendor, making it difficult to combine a fast vision model (like YOLO) with a realtime reasoning model (like Gemini) in one call chain.
What This Does
Vision-Agents is a Python framework for building multi-modal agents that watch, listen, and speak in real time. The core logic lives in agents-core/visionagents/core/, with clean separation between edge/ (transport), llm/, stt/, tts/, and processors/ (vision models). The Agent class in core/agents/agents.py wires these together, and core/runner/runner.py provides an HTTP server for deployment.
The framework supports multiple LLM providers with native SDK methods (OpenAI, Gemini, Claude), MCP integration via core/mcp/, and a plugin system documented in plugins/README.md. The examples/ directory contains eight working reference implementations, from a simple agent (01simpleagentexample/) to a Prometheus-monitored deployment (06prometheusmetricsexample/).
How To Use It
Setup: The project uses pyproject.toml and uv.lock files throughout, so uv is the expected package manager. Install the core package from agents-core/ and run any example from its directory:
cd agents-core uv sync cd ../examples/01simpleagentexample uv sync
Configuration: Each example has an .env.example file (e.g., examples/.env.example) listing required keys. You will need API keys for your chosen LLM provider, Stream Video credentials (or another edge provider), and provider keys for STT/TTS. Copy the .env.example to .env and fill in the values.
Running it: Each example is a standalone Python script. For example, examples/02golfcoachexample/golfcoachexample.py is the entry point for the golf coach. The 08agentserverexample/ shows how to run the agent as an HTTP server, and 07deployexample/ includes Dockerfiles and Helm charts for production deployment.
Real-World Use
A sports coaching system is the clearest use case. The golf example combines a YOLO pose model (YOLOPoseProcessor) running at 10 FPS with a Gemini realtime LLM. The agent watches the user's swing, detects pose issues, and gives spoken feedback. The same pattern applies to security monitoring, drone inspection, or physical therapy — any scenario where you need low-latency vision analysis with conversational output.
Code Health & Issues
Med - No lockfile for core package: agents-core/pyproject.toml declares dependencies without a uv.lock, so builds are not fully reproducible. Examples do have lockfiles. Med - Testing is thin: Only 8 test files across the entire repo, with llmtest.py and manualtest.py suggesting some paths are only manually verified. The run_tests.yml workflow exists, but coverage is unclear. Low - GPU dependency hardcoded: The golf example uses device="cuda" in the YOLO processor, which will fail on CPU-only machines without clear fallback. Low - Documentation is AI-generated: The docs/ai/instructions/ directory contains per-module guidance that reads like generated documentation — useful, but verify against actual behavior. Good signs: CI is configured (.github/workflows/ci.yml), pre-commit hooks exist, and the codebase has clean module separation with consistent patterns across events.py files.
The Bottom Line
This is a well-architected framework for a genuinely hard problem, backed by Stream's edge network for the latency story. The example-driven approach makes it practical to adopt, and the provider-agnostic design avoids vendor lock-in. The main concerns are thin test coverage and missing lockfile reproducibility for the core package. Best suited for teams building real-time video AI products who need a proven starting point rather than a from-scratch integration.