The Problem

On‑prem GPU clusters need a low‑overhead way to surface per‑GPU utilization, temperature, power draw and active processes without installing a full monitoring stack. Existing solutions are either heavyweight (Prometheus + Grafana) or require manual instrumentation.

What This Does

gpu-hot ships a single Docker image that runs a Python web server (app.py) exposing a JSON API and a WebSocket stream. The server gathers metrics via NVIDIA’s NVML library (core/monitor.py) and falls back to nvidia‑smi (core/nvidia_smi_fallback.py) when NVML is unavailable. A React front‑end (static files under static/ and templates/) renders real‑time charts. Multi‑node aggregation is handled by core/hub.py, which contacts peer nodes over HTTP and merges their metric payloads.

Key files:

  • app.py – FastAPI entry point, defines api_gpu_data, api_version, and the root HTML route.
  • core/monitor.py – GPUMonitor class implements the main polling loop (monitor_loop).
  • core/metrics/collector.py – MetricsCollector builds the per‑GPU dictionary; uses helpers in core/metrics/utils.py (safe_get, decode_bytes, to_mib, to_watts).
  • core/hub.py – Hub class discovers peer URLs (NODE_URLS) and aggregates results for hub mode.
  • static/js/*.js – WebSocket client, chart rendering and UI glue.

How It Is Wired

Execution starts with the FastAPI app created in app.py. An HTTP GET on /api/gpu-data calls api_gpu_data, which instantiates a GPUMonitor (or Hub when GPU_HOT_MODE=hub).

  • GPUMonitor.__init__ → core/monitor.py → monitor_loop (runs in a background thread).
  • Inside the loop, GPUMonitor.get_gpu_data → MetricsCollector.collect_all (core/metrics/collector.py).
  • collect_all builds the payload by invoking a set of _add_* helpers; each helper calls safe_get (the most‑used internal function, 23 call sites) and conversion helpers (decode_bytes, to_mib, to_watts).
  • The resulting dict is returned to api_gpu_data, serialized to JSON and pushed to connected browsers via the WebSocket endpoint defined in core/handlers.py.

In hub mode, api_gpu_data creates a Hub instance (core/hub.py). Hub.get_cluster_data iterates over NODE_URLS, performs an outbound HTTP request (session.get in api_version shows the pattern) and merges each node’s metric dict. The hub therefore has the widest blast radius: any change to its request handling or aggregation logic propagates to all downstream nodes.

The front‑end (static/js/app.js → socket-handlers.js) opens a WebSocket to /socket.io/, receives the JSON payload, and hands it to chart managers (chart-manager.js, gpu-cards.js). The latter file is oversized (942 lines) and contains duplicated 6‑line blocks across several JS modules, increasing maintenance risk.

No circular imports were detected; the import graph consists of 41 internal modules with 26 edges, indicating a relatively flat dependency structure.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/gpu-hot
cd gpu-hot

# Build and run the container (requires NVIDIA Container Toolkit)
docker compose up --build -d

Environment variables (documented in the README and core/config.py):

# Single‑node
NVIDIA_VISIBLE_DEVICES=0,1          # GPUs to expose (default: all)
UPDATE_INTERVAL=0.5                # NVML poll interval (seconds)

# Hub mode
GPU_HOT_MODE=hub
NODE_URLS=http://node1:1312,http://node2:1312
NODE_NAME=$(hostname)               # Display name in UI

The service listens on port 1312 (configurable in core/config.py). Access the dashboard at http://localhost:1312 or query the JSON API:

curl http://localhost:1312/api/gpu-data
curl http://localhost:1312/api/version

Real‑World Use

A data‑science team runs the image on each GPU‑enabled host. A central “hub” container aggregates the metrics and feeds a Grafana dashboard via the WebSocket stream, allowing the team to spot throttling or memory pressure instantly without adding a full Prometheus scrape target.

Code Health & Issues

  • HIGH – GitHub Actions reference mutable tags (docker/setup-buildx-action@v3, etc.). Pin to commit SHAs to avoid supply‑chain drift.
  • HIGH – CI workflow (.github/workflows/publish.yml) never executes the test suite despite 29 test files. Add a test step.
  • MEDIUM – No Dependabot/Renovate configuration; vulnerable dependencies remain unpatched.
  • MEDIUM – Dockerfile uses mutable base image nvidia/cuda:12.2.2-runtime-ubuntu22.04. Pin by digest.
  • MEDIUM – No container user defined; container runs as root. Add a non‑root USER.
  • MEDIUM – No dependency‑vulnerability scan in CI; add dependency-review-action or osv-scanner.
  • LOW – Workflow jobs lack timeout-minutes; set a reasonable limit to prevent overlapping runs.

Additional observations: the repository ships a requirements.txt without a lockfile, making reproducible builds non‑deterministic; the JavaScript side contains deep nesting (max indentation depth 8) and duplicated logic, increasing cognitive load for future contributors.

The Bottom Line

gpu-hot provides a lightweight, containerized GPU‑monitoring stack that works out‑of‑the‑box for single‑node and clustered deployments. The codebase is functional but suffers from high‑complexity sections, missing CI safeguards, and security‑hardening gaps that should be addressed before production use. It is best suited for teams comfortable with Docker and willing to invest in modest refactoring and CI improvements.