ai for k8s

The Problem

Organizations that own GPU‑enabled hardware want a turnkey AI platform that runs on bare metal, not a cloud‑hosted demo. They need GPU drivers, storage, inference APIs, training pipelines, and notebooks all orchestrated by a single repository, without hypervisor overhead.

What This Does

ai-for-k8s delivers a self‑hosted PaaS that ships the entire stack—GPU drivers, Rook/Ceph storage, Argo Workflows, vLLM, KServe, and Jupyter—on Talos Linux. Key entry points are the bootstrap/ scripts that bootstrap a Talos cluster and install ArgoCD. Once the cluster is up, clusters/<tier>/ contains the tier‑specific ArgoCD application manifests that sync the rest of the stack.

How It Is Wired

Responsibility	File / Folder	What It Touches
Cluster bootstrap	`bootstrap/gen-config.sh`, `bootstrap/apply-config.sh`, `bootstrap/bootstrap-cluster.sh`	Generates Talos config, applies it via `talosctl`, installs Cilium, ArgoCD, and the GPU driver extensions.
ArgoCD source	`clusters/<tier>/apps/*.yaml`	Each file is an ArgoCD Application that points to the Git repo (e.g., `clusters/max/apps/10-platform.yaml`).
Platform services	`platform/argocd/`, `platform/manifests/`	Deploys Cilium, Envoy Gateway, cert‑manager, and monitoring stack.
GPU management	`gpu/argocd/`, `gpu/manifests/`	Installs NVIDIA device plugin, DRA, DCGM, NVSentinel, and configures time‑slicing.
Inference	`inference/argocd/`, `inference/manifests/`	Deploys GIE, KServe, vLLM, LiteLLM, KubeAI, and Ollama workloads.
Training	`training/argocd/`, `training/manifests/`	Installs Kubeflow Trainer v2, Kueue TAS, JobSet, and NCCL.
Storage	`storage/argocd/`, `storage/manifests/`	Deploys Rook‑Ceph, Ceph RGW, SeaweedFS, and local‑path provisioners.
Data pipelines	`data/argocd/`, `data/manifests/`	Argo Workflows for ingest→embed→upsert into Qdrant or Milvus.
Notebooks	`notebooks/argocd/`, `notebooks/manifests/`	Deploys Z2JH JupyterHub with OIDC and idle culling.

Execution starts with bootstrap/bootstrap-cluster.sh. It calls gen-config.sh → apply-config.sh, which runs talosctl apply-config. The resulting cluster runs Talos with GPU extensions baked in. ArgoCD then pulls the clusters/<tier>/ manifests, syncing each component. Each component deploys via its own Helm chart or Kubernetes manifests, so a change to a single YAML file propagates only that service.

The graph has a single entry point and a fan‑out to 6 application projects (platform, gpu, storage, inference, training, notebooks). The only non‑idempotent step is the initial Talos image build; thereafter all changes are GitOps driven.

How To Use It

Clone the repo ``bash git clone https://github.com/moses-y/ai-for-k8s cd ai-for-k8s ``

Configure the repo host Replace YOUR_ORG in all YAML files: ``bash grep -rl 'YOUR_ORG' --include='*.yaml' . | xargs sed -i '' 's|YOUR_ORG/k8s-for-ai|yourorg/k8s-for-ai|g' ``

Bootstrap a Talos cluster ``bash ./bootstrap/bootstrap-cluster.sh ``

This script expects talosctl, kubectl, helm, argocd in $PATH. It will generate a Talos config, apply it, install Cilium, and deploy ArgoCD.

Choose a tier Edit clusters/<tier>/apps/root-app.yaml to point to the desired tier (e.g., clusters/minimal). ArgoCD will sync all applications under that tier.

Verify ``bash kubectl get pods -A ``

All services should be running; the inference API is available at the Envoy gateway.

Real‑World Use

A data‑science team boots a minimal cluster, then runs argo workflows submit data/argocd/argo-workflows.yaml. The workflow pulls raw data, chunks it, embeds with a HuggingFace model, and upserts vectors into Qdrant. A subsequent inference request to /v1/chat/completions is routed by Envoy to the LiteLLM service, which forwards to the underlying vLLM model.

Code Health & Issues

Missing CI/CD – No .github/ or GitLab CI config; deployments are manual.
No LICENSE – Usage rights unclear.
Secret handling – ADR-008 documents secrets; platform/argocd/external-secrets.yaml references ExternalSecrets, but no config is committed.
Tests – Two test files present, but no automated test gate.
Documentation – 25 Markdown files, but core installation steps are in README.md only.

The Bottom Line

The repository is a solid, GitOps‑driven foundation for a self‑hosted AI platform on bare‑metal GPUs. It scales from a single CPU box to a multi‑node datacenter cluster, with clear separation of concerns across tiers. The lack of CI/CD, license, and secret vetting are the main gaps. Organizations with their own Talos‑enabled infrastructure and a need for an on‑prem AI stack will benefit, but they should add their own CI pipeline and licensing before production use.