The Problem

Deploying a low‑latency voice‑agent service requires wiring WebRTC transport, VAD‑driven turn‑taking, streaming STT/LLM/TTS, and optional telephony—all while keeping the media path under the developer’s control. Building that stack from scratch forces teams to re‑implement ICE, RTP, Opus encoding, and interruption handling, which introduces bugs and latency spikes.

What This Does

streamcore-server implements the media runtime in Go.

  • The core server starts in main.go (entry point main) and spins up the TURN server (internal/turn/server.go) and the WHIP endpoint (internal/signaling/handler.go).
  • Audio handling lives in internal/audio/opus.go (Opus encoder/decoder) and internal/audio/rtp.go (RTP packetisation).
  • Session orchestration, VAD, and barge‑in logic are in internal/pipeline/* (e.g., pipeline.go, inbound.go, realtime.go).
  • LLM, STT, and TTS adapters are under internal/llm/, internal/stt/, and internal/tts/ (e.g., openai.go, deepgram.go, cartesia_ws.go).
  • Plugins (gmail, math, vision, etc.) are loaded from plugins/ via the external plugin manager (internal/plugin/external.go).
  • Optional SIP bridge and ESP32 support are provided by separate modules not shown in the call graph but reachable through the plugin system.

How It Is Wired

Execution begins at main.go:25 → main. main loads the configuration (internal/config/config.go → Load), registers plugins (loadPlugins), and starts the HTTP server that routes WHIP POST/DELETE to handleWHIPPost / handleWHIPDelete (internal/signaling/handler.go).

handleWHIPPost creates a new peer (internal/peer/peer.go → New), which in turn builds an Opus encoder/decoder and a data‑channel for realtime events. The peer registers callbacks that forward inbound RTP packets to the pipeline (internal/pipeline/pipeline.go → HandleDataChannelMessage).

The pipeline (pipeline.go) calls newRealtimeAudioQueue, which invokes VAD (internal/vad/vad.go) to decide turn boundaries. When a turn is closed, internal/pipeline/agent.go triggers synthesizeSentences, which calls the TTS client (internal/tts/cartesia_ws.go → SynthesizeStream). The TTS client opens a WebSocket to Cartesia, streams audio chunks, and writes them back to the RTP stream.

LLM interaction occurs via internal/llm/openai.go → Chat, which is invoked from internal/pipeline/agent.go when a user turn is ready. The call chain main → Load → loadPlugins → New → handleWHIPPost → New → pipeline → Chat touches external services exactly twice: once for a model inference (OpenAI) and once for a TTS WebSocket.

The most widely referenced functions are Load (config), Add/Close (transcript handling), and sessionUpdate (realtime state). Load is called from 38 distinct locations, giving it the largest blast radius; changes here affect every request that reads configuration or plugin metadata.

No circular import cycles were detected, and the internal call graph contains 739 resolved edges, indicating a fairly flat dependency structure. However, deep nesting (up to 7 levels) appears in inbound.go, pipeline.go, and cartesia_ws.go, which can make future modifications harder to trace.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/streamcore-server
cd streamcore-server

# Build the Go binary (requires Go 1.22+)
go build -o streamcore .

# Or run directly
go run main.go

The server reads configuration from config.toml.example (copy to config.toml and edit). Required keys include:

  • whip.secret – JWT secret for optional WHIP auth.
  • stt.provider / tts.provider – names matching the adapters under internal/stt/ and internal/tts/.

For the bundled Python STT/TTS helpers:

cd external/vibeVoice/vibeVoiceAsr
pip install -r requirements.txt
python server.py &
cd ../../vibeVoiceTTS
pip install -r requirements.txt
python server.py &

These expose local HTTP endpoints that the Go runtime can call if configured.

To launch the full Docker stack (includes Caddy proxy and optional Supabase DB):

docker build -t streamcore .
docker compose -f infrastructure/aws/ec2/docker-compose.yml up -d

Real‑World Use

A SaaS voice‑assistant product can point its client SDKs (React, Python, Rust) at the WHIP endpoint (POST /whip). Audio is streamed over Opus/RTP, VAD splits turns, the server forwards the user transcript to a hosted LLM (e.g., OpenAI), receives a response, streams TTS back to the client, and reports per‑turn latency via the data‑channel. Plugins such as plugins/plugins/gmail can be invoked as tool calls from the LLM without leaving the media runtime.

Code Health & Issues

  • High – clarity – duplicated 6‑line blocks across 14 files (e.g., vibeVoiceAsr/server.py, vibeVoiceTTS/server.py, internal/llm/ollama.go, internal/llm/openai.go). Refactor into shared helpers.
  • Medium – cognitive_load – deep nesting (max depth 7) in internal/pipeline/inbound.go, pipeline.go, cartesia_ws.go. Flatten with early returns.
  • Medium – resilience – broad except: in plugins/plugins/time-get/main.py. Replace with specific exception handling.
  • Medium – cognitive_load – high branching density (46 branches over 156 lines) in several pipeline files. Consider strategy tables or smaller functions.

No missing license, CI, or Dockerfile; tests are present (27 files). The repository includes a lockfile (go.sum, package-lock.json) and no committed secrets.

The Bottom Line

streamcore-server delivers a production‑grade, Go‑native media layer for real‑time voice agents, with clear separation of transport, VAD, and AI integration. The codebase is well‑tested but suffers from duplicated snippets and deeply nested logic that will increase maintenance cost. Teams needing tight control over latency and media handling will find it usable, provided they allocate effort to clean up the identified hotspots.