Vision Agents - Moses Yebei

Abstract

Open Vision Agents by Stream. Build Vision Agents quickly with any model or video provider. Uses Stream's edge network for ultra-low latency.

The Problem

Real-time video AI agents are hard to build because the pipeline spans video ingestion, object detection, speech, and an LLM, and each hop adds latency. Most frameworks treat video as an afterthought or lock you into one model provider. This repo solves that by giving you a Python core that wires video/audio tracks, vision processors (YOLO, Roboflow), and pluggable LLM/STT/TTS providers together over Stream's low-latency edge network, with a claimed sub-30ms audio/video path.

What This Does

agents-core/ is the framework: vision_agents/core/agents/agent_launcher.py owns the agent lifecycle, agents.py the turn loop, runner/runner.py serves agents over HTTP, and edge/sfu_events.py is the generated protocol layer for Stream's SFU. Provider integrations live in plugins/ (openai, gemini, cartesia, deepgram, elevenlabs, ultralytics). Seven working examples in examples/ show the pattern end to end, from a golf coach (YOLO pose + Gemini realtime) to a security camera that posts "WANTED" posters.

How It Is Wired

Execution starts at start in agent_launcher.py:147 (reaches 44 functions, nothing else calls it) or run in runner.py:83. The example entry point plugins/cartesia/example/main.py calls join_call, which reaches 137 functions — the widest reach in the repo. The shortest path to an external effect is start -> warmup -> _warmup_agent, which hits a model via agent.llm.warmup; the join path reaches the network through simple_response via client.post in 2 hops. Across the repo, 54 functions call a model, 27 make network calls, 24 touch the filesystem, 1 hits a database.

The blast-radius hubs: edge/sfu_events.py (241 functions, called from 12 files), agent_launcher.py (called from 36 files), and agents.py (67 functions, called from 25). agents.py sits in a 34-module circular-import cycle with instability 0.93, so changing it ripples widely. The most-called functions are wait (47 call sites), simple_response (31), and upsert_message (26) — these are the seams to change carefully.

How To Use It

Setup uses uv (examples ship uv.lock; agents-core/pyproject.toml is the package manifest). Copy .env.example to .env with your provider keys, then run an example:

uv sync
cp .env.example .env
uv run examples/02_golf_coach_example/golf_coach_example.py

Deployment options exist in examples/07_deploy_example/ (Dockerfile, docker-compose, Helm chart). The README documents the Agent(...) constructor pattern with edge, llm, processors, tts, stt arguments.

Real-World Use

A security-camera agent from examples/05_security_camera_example/: a SecurityCameraProcessor runs YOLOv11 at 5 fps to detect package theft, gemini.LLM("gemini-2.5-flash-lite") reasons about frames, and deepgram.STT()/elevenlabs.TTS() handle voice. The agent posts generated posters to X automatically — a vision pipeline, not a chatbot.

Code Health & Issues

Static analysis (not opinion) found 121 issues: 58 high (27 deep-nesting sites, 27 circular-import members, duplicated code blocks — 939 repeated 6-line blocks across 108 files, likely the generated protobuf layer), 61 medium, 2 low. The SDLC audit found 7 medium, 1 low: un-pinned GITHUB_TOKEN permissions in .github/workflows/ci.yml, no Dependabot, unpinned base image (python:3.13-slim), no dependency vulnerability scan, 32MB GIFs in assets/ (no LFS), persist-credentials left on in release.yml, and no non-root USER in the deploy Dockerfile. Tests and CI exist (106 test files, GitHub Actions).

The Bottom Line

A credible, production-oriented framework for real-time vision agents with strong provider coverage and working deployment examples. The circular imports and generated-code duplication make agents.py and sfu_events.py costly to modify, and CI hygiene needs tightening. Use it if you want a working vision-agent stack fast; expect to refactor if you need deep customization of the core.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

578 files

Python340
Markdown87
TOML58
YAML19
JSON2
Shell1

DockerKubernetesTransformersYOLOCUDA

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["plugins 210 modules"] g1["agents-core 76 modules"] g2["tests 28 modules"] g3["examples 24 modules"] g4["(root) 2 modules"] g0 -->|7| g4

Findings

LowDependencies declared without a lockfile - non-reproducible builds agents-core/pyproject.toml

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

medium7

low1

Declare least-privilege permissions for GITHUB_TOKENmedium3 occurrences

.github/workflows/ci.yml

3 workflow(s) declare no permissions, 1 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Enable Dependabot or Renovatemedium

31 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

examples/07_deploy_example/Dockerfile

python:3.13-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Move large binaries to Git LFS or out of the repositorymedium5 occurrences

assets/demo_gifs/geoguesser.gif

5 blobs over 5MB: geoguesser.gif 32.2MB, mirage.gif 29.7MB, golf.gif 20.5MB

One repository here carries twenty blobs over 5MB including a 10.9MB spreadsheet, so every clone and every CI checkout pays for data nobody diffs.

Fix: Track those extensions with LFS, or move datasets to object storage and fetch them in a setup step.

Set persist-credentials: false on checkoutmedium

.github/workflows/release.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

examples/07_deploy_example/Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Set timeout-minutes on the workflow jobslow3 occurrences

.github/workflows/ci.yml

3 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.