aisix

Abstract

Open-source AI gateway for LLMs & AI agents, built in Rust. One OpenAI-compatible API for OpenAI, Anthropic, Gemini, Bedrock & more — routing, guardrails, caching, rate limits, observability.

Repository: aisix Clone:

git clone https://github.com/moses-y/aisix

The Problem

Teams that expose multiple LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure OpenAI, etc.) must run separate adapters, duplicate auth, and stitch together rate‑limit, caching and observability logic. Maintaining that plumbing in each service creates operational drift and hidden cost.

What This Does

aisix provides a single, OpenAI‑compatible HTTP API that fronts all supported providers. The gateway is a static Rust binary (cargo build --release) that reads declarative resources from resources.yaml or an etcd cluster and enforces:

routing & fail‑over (crates/aisix-gateway/src/bridge.rs)
API‑key auth & provider‑key mapping (crates/aisix-admin/src/provider_keys_handlers.rs)
guardrails such as moderation, PII redaction (crates/aisix-guardrails/src/*.rs)
per‑model caching (crates/aisix-cache/src/*.rs) and rate limits (crates/aisix-core/src/models/rate_limit.rs)
Prometheus / OTLP observability (crates/aisix-obs/src/*.rs)

The admin REST interface lives in crates/aisix-admin, the core data model in crates/aisix-core, and each provider implementation under crates/aisix-provider‑*.

How It Is Wired

Layer	Entry point	Primary responsibilities	Blast‑radius
Binary	`crates/aisix-gateway/src/lib.rs` (invoked from `bench/pgo-training/trainer/src/main.rs` for benchmarks)	Parses CLI args, loads config (`crates/aisix-core/src/config.rs`), starts HTTP server (`crates/aisix-gateway/src/hub.rs`)	Medium – changes affect all request handling
Routing hub	`crates/aisix-gateway/src/hub.rs` → `crates/aisix-gateway/src/bridge.rs`	Selects provider based on request, builds upstream HTTP (`crates/aisix-gateway/src/upstream_http.rs`)	High – 197 modules import `tests/e2e/src/harness/index.ts` which mirrors hub behavior; churn here propagates widely
Provider bridge	`crates/aisix-provider-*/src/bridge.rs` (e.g., `aisix-provider-openai`)	Translates OpenAI‑compatible payloads to provider‑specific wire format (`src/wire.rs`) and back	Medium – each provider isolated
Cache layer	`crates/aisix-cache/src/cache.rs` → `semantic_redis.rs`	In‑memory + Redis backing; key generation in `crates/aisix-cache/src/key.rs` (deep nesting, 8‑level indent)	Low – limited to cache calls
Guardrails	`crates/aisix-guardrails/src/index.rs`	Calls moderation APIs (e.g., OpenAI, Presidio) before upstream request	Medium – invoked for every request when enabled
Observability	`crates/aisix-obs/src/lib.rs` → `metrics.rs`, `otlp.rs`	Emits Prometheus counters, OTLP spans	Low – side‑effect only

The import graph contains 215 internal modules and 233 edges, with no circular dependencies. The most connected module is the test harness tests/e2e/src/harness/index.ts (197 dependents), indicating that the test harness mirrors core routing logic and should stay stable.

How To Use It

Build the binary (requires Rust 1.70+):

   cd aisix
   cargo build --release

Configure a minimal config.yaml (see config.example.yaml). Example excerpt:

   resources_file: /etc/aisix/resources.yaml
   proxy:
     addr: "0.0.0.0:3000"
   admin:
     enabled: false
   observability:
     metrics:
       prometheus:
         enabled: true
         addr: "0.0.0.0:9090"

Provider keys and models are declared in resources.yaml (see assets/aisix-architecture.svg for the schema).

Run the gateway:

   ./target/release/aisix-gateway --config config.yaml

The process listens on the address in proxy.addr and serves the OpenAI‑compatible API.

Optional Docker – a ready‑to‑run image can be built with the provided Dockerfile:

   docker build -t aisix .
   docker run -p 3000:3000 -v $(pwd)/resources.yaml:/etc/aisix/resources.yaml aisix

No additional scripts are required; the repository supplies a complete CI pipeline (.github/workflows/ci.yml) and a benchmark harness for performance testing.

Real‑World Use

A SaaS platform that needs to expose both OpenAI GPT‑4 and Anthropic Claude‑3 can point its existing client SDKs at the AISIX endpoint. The gateway handles provider‑specific auth, enforces a per‑user token budget, caches repeated completions, and streams results via SSE—all without code changes in the service layer.

Code Health & Issues

Measured findings (154 total)

High – Hub module: tests/e2e/src/harness/index.ts is depended on by 197 modules. Keep it minimal; volatile logic should be moved elsewhere.
High – Deep nesting: crates/aisix-admin/src/openapi.rs, crates/aisix-cache/src/key.rs, crates/aisix-guardrails/src/aliyun_ai_guardrail.rs have indentation depth 8, reducing readability. Refactor with early returns or extracted helpers.
High – Duplicated blocks: Six‑line shell snippets repeated in 304 files (e.g., bench/onthebench/run‑*.sh). Consolidate into a shared script.
High – Oversized files: crates/aisix-admin/src/lib.rs, crates/aisix-admin/src/openapi.rs, crates/aisix-core/src/config.rs each exceed 1 250 lines. Split by logical concern.

Code‑health audit (8 findings)

High – Pin GitHub Actions to commit SHAs (.github/workflows/*).
High – Remove continue-on-error from correctness steps (.github/workflows/ci.yml).
Medium – Declare least‑privilege permissions for GITHUB_TOKEN.
Medium – Add Dependabot (.github/dependabot.yml).
Medium – Pin Docker base images by digest.
Medium – Add a dependency‑vulnerability scan to CI.
Medium – Set persist-credentials: false on checkout.
Low – Define timeout-minutes for workflow jobs.

Other hygiene

Test suite present (241 test files) and executed via GitHub Actions.
License (LICENSE) and lockfile (Cargo.lock) are included.
Secret‑shaped PEM files exist under crates/aisix-provider-vertex/test-fixtures/; they are test fixtures, not production secrets, but should be reviewed for accidental commit.

The Bottom Line

aisix delivers a production‑grade, Rust‑native gateway that consolidates LLM provider access, guardrails, caching and observability behind a single OpenAI‑compatible API. The codebase is functional and well‑tested but suffers from a few high‑impact hotspots—particularly the central test harness and several deeply nested, oversized modules. Addressing those, together with the straightforward CI hardening recommendations, will improve maintainability and security. The project is a solid foundation for teams that need a unified LLM ingress point and are comfortable working in Rust or container environments.