The Problem
GPU‑intensive workloads need a reliable way to inspect which processes occupy which NVIDIA devices, monitor utilization in real time, and integrate that data into observability stacks. Existing tools (nvidia‑smi, gpustat) give static snapshots or limited UI, making it hard to troubleshoot contention in containerised or multi‑user environments.
What This Does
nvitop provides an interactive terminal UI (see nvitop/tui/) that lists devices, processes, memory, power, and temperature, updating live. The core API lives under nvitop/api/ (e.g., device.py, process.py, libnvml.py) and is reusable by other Python code.
A complementary exporter (nvitop-exporter/) reads the same API and exposes Prometheus metrics via nvitopexporter/exporter.py. Sample Grafana dashboards are bundled in nvitop-exporter/grafana/. The CLI entry points are nvitop/main.py and nvitop/cli.py, while the exporter can be started via nvitop-exporter/nvitopexporter/main.py.
How To Use It
Setup
Install the library and CLI from PyPI (recommended) pip install nvitop
Or install from source
python -m pip install -e . # runs setup.py / pyproject.toml
For the exporter container: docker build -t nvitop-exporter -f nvitop-exporter/Dockerfile . docker run -p 8000:8000 nvitop-exporter
The repository also includes a Dockerfile for the UI (Dockerfile) if you prefer an isolated environment.
Configuration
No mandatory config files are required for basic operation. The exporter reads optional environment variables documented in nvitop-exporter/README.md (e.g., NVITOPEXPORTERPORT). The UI can be customized through command‑line flags described in nvitop/cli.py.
Running it
Interactive UI
python -m nvitop # invokes nvitop/main.py or simply nvitop # after pip install adds console script
Keybindings are defined in nvitop/tui/library/keybinding.py (e.g., q to quit, h for help).
Prometheus Exporter
python -m nvitopexporter # runs nvitop-exporter/nvitopexporter/main.py exposes metrics at http://localhost:8000/metrics
Real‑World Use
A CI node running multiple TensorFlow jobs can launch nvitop on the host to spot GPU contention before a job stalls. Simultaneously, nvitop-exporter runs as a sidecar in the same pod, feeding metrics to a central Prometheus server; Grafana dashboards (in nvitop-exporter/grafana/) visualize per‑process GPU usage across the fleet.
docker‑compose snippet for exporter sidecar services: trainer: image: my‑tf‑job runtime: nvidia exporter: build: ./nvitop-exporter ports: ["8000:8000"] environment: NVITOPEXPORTERPORT=8000
Code Health & Issues
Low – Missing lockfiles – requirements.txt and pyproject.toml are present but no poetry.lock/pipfile.lock; reproducible builds depend on exact PyPI versions. Medium – No test suite – No tests/ directory and CI workflow (.github/workflows/build.yaml) does not run unit tests, leaving core paths unverified. Low – Limited CI linting – Lint workflow exists (lint.yaml) but no coverage or static analysis for security (e.g., bandit). Low – Dockerfile assumes NVIDIA runtime – No explicit ARG to configure runtime; containers will fail on non‑GPU hosts without modification. Low – Documentation breadth – API docs are generated (docs/source/api/*.rst) but some modules (e.g., callbacks/) lack usage examples.
Overall the code follows a clear package structure, uses type hints, and separates UI, API, and exporter concerns cleanly.
The Bottom Line
nvitop delivers a practical, Python‑native UI and a Prometheus exporter for NVIDIA GPU monitoring, suitable for DevOps teams that already use Python and container orchestration. The main drawbacks are the absence of automated tests and lockfiles, which raise confidence and reproducibility concerns for production deployments. Teams comfortable adding their own test coverage can adopt it quickly; otherwise, evaluate the risk before integrating into critical pipelines.