The Problem
Running many autonomous agents on shared Kubernetes clusters requires isolation that standard containers cannot provide. Agents are idle most of the time, so containers waste memory, and untrusted code can access credentials if a pod is compromised. A runtime that can snapshot idle agents and resume them quickly while keeping each agent in its own micro‑VM is needed.
What This Does
- Zero‑trust sandboxing – every agent runs inside a dedicated gVisor or Cloud Hypervisor micro‑VM, using the Go package in
vendor/github.com/myzhan/gomq/zmtp. - Snapshotting & fast resume – the core logic lives in the
internal/snapshotpackage, which writes a VM state to disk and restores it in under 500 ms. - Kubernetes‑native – manifests in
site/and acmd/entry point let the runtime register itself with the API server, expose a REST API, and publish Prometheus metrics. - SPIFFE identity – the server creates a short‑lived X.509 certificate for each running agent in
internal/identity, ensuring that egress traffic can be authenticated.
How It Is Wired
- Entry point –
._LICENSES/github.com/myzhan/gomq/server.gostarts the HTTP server. It importsinternal/apifor routing,internal/controllerfor business logic, andinternal/identityfor SPIFFE handling. - API layer –
internal/api/handlers.gomaps routes to controller functions. - Controller hub –
internal/controller/controller.gois the central dispatcher. It callsinternal/snapshot/manager.goto create or restore a VM andinternal/monitor/metrics.goto expose metrics. - Snapshot engine –
internal/snapshot/manager.gousesgomq/zmtpto talk to the micro‑VM runtime and writes snapshot files to the path defined insite/config.yaml. - Side effects – the controller writes to the Kubernetes API via
client-go(imported invendor/k8s.io/client-go), logs to stdout, and updates Prometheus metrics. The widest blast radius is the controller, which has 12 direct callers.
How To Use It
# 1) Clone
git clone https://github.com/moses-y/substrate
# 2) Build the server
cd substrate
go build ./cmd/... # builds binaries for each main package
# 3) Configure
# Create /etc/substrate/config.yaml # example in site/
export SUBSTRATE_CONFIG=/etc/substrate/config.yaml
export SPIFFE_TRUST_DOMAIN=example.org
# 4) Run
./cmd/server # starts the HTTP API on :8080
For benchmarking a micro‑VM, build the test image:
docker build -t substrate/atunnel-bench benchmarking/atunnel
Real‑World Use
A self‑hosted inference platform can launch each inference job as an agent. The platform calls POST /agents/start with the agent name, and Substrate resumes the snapshot from disk if the agent is idle, or starts a new micro‑VM if not. This keeps memory usage low while ensuring a 500 ms resume window for the next request.
Code Health & Issues
- Low – potential exposed secrets in
vendor/github.com/subosito/gotenv/.env.
No other security or lint issues were reported by the static analysis. The repository contains 1237 test files, but coverage percentages are not provided. CI is configured with Travis CI, and the make target make test runs all tests.
The Bottom Line
Agent Substrate delivers a robust, micro‑VM‑based runtime for large‑scale agent workloads on Kubernetes. The Go codebase is well‑structured around a clear controller hub, and the built‑in snapshotting provides fast idle‑time scaling. It is ready for production use in environments that already run Kubernetes and can supply SPIFFE identities and appropriate network policies.