realtime phone agents course

Abstract

Build realtime AI voice agents using FastRTC for low-latency streaming, Superlinked for vector search, Twilio for live phone calls, and Runpod for scalable GPU deployment.

realtime-phone-agents-course Clone: https://github.com/moses-y/realtime-phone-agents-course

The Problem

Building a production‑grade real‑time voice‑agent system that must simultaneously handle inbound/outbound Twilio calls, low‑latency STT/TTS streaming, vector‑based property search, and GPU‑accelerated deployment is non‑trivial. The repository bundles four major frameworks (FastRTC, Superlinked, Twilio, Runpod) but lacks test coverage, lock‑file reproducibility, and hardened security defaults, making incremental changes risky for a technical team.

What This Does

The codebase implements a full voice‑agent pipeline:

Twilio integration – inbound calls land in src/realtime_phone_agents/api/main.py (main entry point) and are handed to the FastRTC agent via the lifespan context manager.
STT – two back‑ends are available: src/realtime_phone_agents/stt/groq/whisper.py (Groq‑hosted) and src/realtime_phone_agents/stt/runpod/faster_whisper/model.py (local Fast‑Whisper on Runpod).
TTS – three synthesis paths: src/realtime_phone_agents/tts/togetherai/model.py, src/realtime_phone_agents/tts/runpod/orpheus/model.py, and a local Kokoro implementation (src/realtime_phone_agents/tts/local/kokoro.py).
Property search – Superlinked vector index (src/realtime_phone_agents/infrastructure/superlinked/service.py) ingests data/properties.csv and serves queries from the API route src/realtime_phone_agents/api/routes/superlinked.py.
Avatars & background effects – avatar registry (src/realtime_phone_agents/avatars/registry.py) loads YAML definitions; keyboard‑sound effects live in src/realtime_phone_agents/background_effects/keyboard.py.

The internal call graph contains 125 resolved call edges; the most‑connected functions are stream_tts (6 callers), stt (5 callers), and FastRTCAgent (3 callers). Execution leaves the process via main → make_call (network) and via model inference in src/realtime_phone_agents/tts/runpod/orpheus/model.py (outbound network call for tokenisation).

How It Is Wired

Entry point	Reaches	Primary outbound effects
`main` (`src/realtime_phone_agents/api/main.py:14`)	34 functions, called from 1 place	`make_call` (Twilio POST), model inference in TTS/STT modules
`lifespan` (`src/realtime_phone_agents/api/main.py:14`)	2 functions, nothing else calls it	Starts/stops GPU pods via Runpod scripts

Key hubs (many depend on them):

src/realtime_phone_agents/agent/fastrtc_agent.py – 21 functions, called from 5 files; defines _process_audio, handler_wrapper.
src/realtime_phone_agents/tts/togetherai/model.py – 11 functions, called from 2 files; contains duplicated 6‑line blocks (see measured findings).
src/realtime_phone_agents/avatars/registry.py – 13 functions, called from 1 file; avatar loading logic.

No circular dependencies were detected; the module graph is a DAG with 63 Python modules and 11 import edges.

How To Use It

Setup

# Clone the repo
git clone https://github.com/moses-y/realtime-phone-agents-course
cd realtime-phone-agents-course

# Create a virtual environment (uv recommended)
uv venv .venv && source .venv/bin/activate

# Install dependencies (pyproject.toml + uv.lock)
uv sync

# Copy example env and fill secrets
cp .env.example .env   # add TWILIO_SID, TWILIO_TOKEN, OPENAI_KEY, GROQ_KEY, RUNPOD_KEY, SUPERLINKED_KEY

Configuration

config.py defines GroqSettings, OpenAISettings, SuperlinkedSettings, QdrantSettings, RunPodSettings – read at runtime by the agent modules.
.env.example lists every required variable; none are committed.

Running

# Start the FastAPI server (uvicorn) – the canonical entry point
uvicorn src.realtime_phone_agents.api.main:app --host 0.0.0.0 --port 8000

# Or launch the Gradio demo (script)
python scripts/run_gradio_application.py

Docker builds are supported via Dockerfile, docker-compose.yml, and the Runpod‑specific Dockerfile.orpheus / Dockerfile.faster_whisper. Use make build or docker compose up --build.

Real‑World Use

A real‑estate agency receives an inbound call via Twilio. The request hits main → lifespan → FastRTCAgent._process_audio, which streams audio to the selected STT model (e.g., stt/runpod/faster_whisper/model.py). The transcript is sent to the LLM agent, which calls get_property_search_service (src/realtime_phone_agents/infrastructure/superlinked/service.py) to query the Superlinked index for matching listings. The LLM’s response is synthesized by tts/togetherai/model.py or tts/runpod/orpheus/model.py and streamed back to the caller through Twilio. Outbound calls follow the same path, initiated by scripts/make_outbound_call.py.

Code Health & Issues

Measured static analysis (pipeline‑generated, 10 findings)

HIGH/cognitive_load – deep nesting x6 in src/realtime_phone_agents/tts/togetherai/model.py, src/realtime_phone_agents/avatars/registry.py, src/realtime_phone_agents/tts/runpod/orpheus/model.py (max indentation depth 10).
HIGH/clarity – duplicated 6‑line blocks across 4 files: src/realtime_phone_agents/background_effects/utils/__init__.py, src/realtime_phone_agents/background_effects/utils/audio_loader.py, src/realtime_phone_agents/tts/runpod/orpheus/model.py, src/realtime_phone_agents/tts/togetherai/model.py (61 repeats).
MEDIUM/resilience – broad except clauses in src/realtime_phone_agents/tts/togetherai/model.py, src/realtime_phone_agents/observability/opik_utils.py, src/realtime_phone_agents/observability/prompt_versioning.py (swallow errors indiscriminately).

Code health audit (SDLC, 9 findings)

High – Pin third‑party GitHub Actions to commit SHAs (.github/workflows); docker/login-action@v3 currently uses a mutable tag.
High – Add a test suite; repository has zero test files (63 source files, no tests).
High – Replace wildcard CORS origin with explicit allow list (src/realtime_phone_agents/api/main.py: allow_origins=["*"] with credentials enabled).
Medium – Declare least‑privilege permissions: contents: read for GITHUB_TOKEN in all 3 deploy workflows (deploy-call-center-dockerhub.yml, etc.).
Medium – Enable Dependabot or Renovate (github-actions and package ecosystems).
Medium – Pin container base image by digest in Dockerfile (python:3.11-slim → python:3.11-slim@sha256<digest>).
Medium – Gate pull requests on a dependency vulnerability scan (add dependency-review-action or osv-scanner).
Medium – Add a non‑root USER to the Docker image (Dockerfile currently runs as root).
Low – Set timeout-minutes on workflow jobs (deploy-call-center-dockerhub.yml et al.; default 6 h risk).

The Bottom Line

This repo provides a functional, end‑to‑end template for real‑time AI phone agents, wiring Twilio, FastRTC, Superlinked, and Runpod‑hosted models into a coherent call‑center flow. The architecture is clear enough to follow, but the codebase suffers from deep nesting, duplicated logic, and absent test/lock‑file hygiene. Teams comfortable with integrating multiple external services and willing to invest in testing and CI hardening will find it a solid starting point; others may need to refactor the high‑nesting modules and add reproducibility guards before deploying to production.