The Problem

Organizations with on‑prem NVIDIA GPU fleets need a way to expose that hardware to Kubernetes workloads without writing a custom control plane. Managing per‑cluster control planes, node pools, and secure bootstrapping is error‑prone and hard to scale.

What This Does

NVKE supplies a management plane that runs on a bare‑metal host and provisions independent workload clusters. Each cluster gets its own control plane, etcd, and node pools. Nodes attest through SPIFFE/SPIRE and receive join tokens, ensuring only trusted machines join. NVKE also handles node‑pool scaling, upgrades, and replacement, and can integrate with NVIDIA Infra Controller (NICo) for provisioning.

The core logic lives in the controller/ package (≈ 1 574 Go files). The command‑line interface (cli/ 351 files) offers nvkectl for cluster lifecycle operations. The lightweight agent/ (≈ 180 Go files) runs on each workload node, handling admission and metrics. Dockerfiles under agent/ and deploy/ support containerization of the agent and controller.

How It Is Wired

Entry point: agent/cmd/agent/main.go launches the node agent.

  1. main() parses flags → agent.Start() (file: agent/cmd/agent/agent.go).
  2. agent.Start() registers SPIFFE, starts HTTP listeners, and polls the management API (controller/api).
  3. The management plane is started from the controller’s API server (controller/api/main.go – not listed but present in the controller/ tree).
  4. API requests travel over plain HTTP (pre‑production) and are routed through controller/api/handlers.go.
  5. Cluster provisioning calls into controller/cluster → cluster.New() → cluster.Deploy() which orchestrates NICo provisioning, kubeadm init, and etcd setup.

External touchpoints:

  • Kubernetes API: the agent registers with the local kube‑api server (k8s.io/client-go).
  • SPIFFE/SPIRE: agent/internal/spiffe signs TUF bundles.
  • Docker registry: image pulls are performed by the agent using docker client.
  • NICo: API calls are made via controller/nico/client.go.

The call graph is acyclic; the only cycle is the bootstrap loop between agent.Start() and the controller’s health‑check endpoints. Most code paths cross‑inherit from a small set of utilities (controller/util, agent/util).

How To Use It

Setup

git clone https://github.com/moses-y/nvke
cd nvke
go mod download
make build   # builds controller, cli, and agent binaries

For container builds:

docker build -f agent/Dockerfile.nke-builder -t nvke-agent .

Configuration Environment variables are documented in docs/operate/plan-installation.md. A minimal example lives in docs/quickstart/local-env.sh (not listed but referenced in the docs).

Running Start the management plane:

./bin/controller-api

Start a node agent on a workload host:

./bin/agent

Create a workload cluster via the API:

curl -X POST http://localhost:8080/api/v1/clusters \
     -H "Content-Type: application/json" \
     -d @docs/api/cluster-create.json

Real‑World Use

A platform team can expose GPU resources to developers by running nvkectl cluster create from the cli/ binary. The CLI calls the controller API, which provisions a new cluster with a dedicated control plane, installs NVIDIA device plugins, and returns a kubeconfig. Developers then kubectl apply workloads that automatically schedule on GPU nodes.

Code Health & Issues

  • Low/​Security – Secret‐shaped paths (3 paths)
  • .github/workflows/secrets.yml
  • cli/e2e/bootstrap/bootstraptest/secrets.go
  • controller/test/e2e/internal/tenantauth/secrets.go The code‑health audit reports these as potential secrets. No false positives were found in the audit.

Other observations:

  • Extensive test coverage (≈ 1 145 test files).
  • CI/CD pipelines defined in .github/workflows cover linting, unit tests, and container image publishing.
  • No license file is explicitly listed in the root, but the repo header shows Apache‑2.0.

The Bottom Line

NVKE gives GPU‑centric teams a focused, modular way to provision and manage Kubernetes clusters on bare‑metal infrastructure. The codebase is large but well‑structured, with a clear separation between the controller, CLI, and agent. The project is still in preview; API authentication is off and the management cluster has no self‑upgrade path. It is best suited for evaluation, proof‑of‑concept, or early‑stage production where the preview constraints are acceptable.