nvitop

The Problem

GPU‑process oversight in heterogeneous clusters is usually split between ad‑hoc nvidia-smi calls, fragmented Python wrappers, and dashboards that require separate exporters. Teams end up maintaining custom scripts just to see who is using a GPU, why utilization spikes, and how to export metrics for monitoring stacks.

What This Does

nvitop is an interactive TUI that visualises NVIDIA‑GPU processes, resource usage, and device topology in real time. It also ships nvitop‑exporter, a Prometheus‑compatible endpoint that translates the same data into metrics for Grafana/Prometheus.

Key components (from the responsibility map):

nvitop/api/device.py – 148 functions, 12 types; defines transmit, receive, __repr__, is_available, driver_version. Called from 13 other files; runs an external command (_parse_cuda_visible_devices via subprocess.check_output).
nvitop/api/process.py – 66 functions, 2 classes; provides add_quotes, command_join, auto_garbage_clean.
nvitop/api/collector.py – 26 functions, 4 classes; snapshot collection (take_snapshots, collect_in_background).
nvitop/tui/tui.py – the TUI entry; max indentation depth 6, control flow hard to follow.
nvitop/api/__init__.py – hub module; 17 other modules import it, making it a high‑blast‑radius point.

The exporter lives in nvitop-exporter/nvitop_exporter/ (cli.py, exporter.py, utils.py) and exposes Prometheus metrics via a simple HTTP server.

How It Is Wired

Execution starts at nvitop/cli.py:272 (main). From there the call graph fans out:

main → parse_arguments → parse_cuda_visible_devices → _parse_cuda_visible_devices (subprocess).
main → cprint → eventually reaches 204 functions.

The most‑connected symbols (call‑edge counts) are nvmlQuery (35 callers), max (27), nvmlCheckReturn (25), bytes2human (23).

Circular import cycle involves nvitop/api/__init__.py, nvitop/tui/library/__init__.py, and nvitop/tui/tui.py – 26 modules are inside cycles, creating instability (instability 0.44 for nvitop/tui/library/__init__).

Hub modules (nvitop/api/__init__.py, nvitop/tui/library/__init__.py, nvitop/tui/tui.py) have 17+ dependents; changes ripple widely.

Duplicated code appears as 292 repeated 6‑line blocks across 43 files (e.g., install-nvidia-driver.sh, nvitop/__init__.py, nvitop/api/__init__.py, nvitop/api/caching.py).

Broad exception handling in nvitop/tui/library/host.py and nvitop-exporter/nvitop_exporter/utils.py swallows errors indiscriminately.

How To Use It

Step	Command / File	Note
Install	`pip install nvitop` (or `pip install -e .` from repo root)	`pyproject.toml` and `setup.py` define the package.
Docker (TUI)	`docker build -t nvitop . && docker run --rm -it --net=host --gpus all nvitop`	Dockerfile uses `ubuntu:latest` (mutable tag – see hygiene).
Exporter	`pip install -e nvitop-exporter && nvitop-exporter/cli.py --port 9090`	Entry point `nvitop-exporter/nvitop_exporter/cli.py:34` reaches 113 functions.
Grafana dashboard	Point Prometheus to `http://<host>:9090/metrics`; dashboard JSON in `nvitop-exporter/grafana/dashboard.json`.

Environment variable CUDA_VISIBLE_DEVICES is respected; the tool auto‑detects devices via subprocess.check_output(['nvidia-smi', '-L']).

Real‑World Use

A ML ops team runs a cluster of 8 A100 nodes. They start nvitop via SSH (ssh user@node nvitop) to instantly see which PID owns each GPU, memory‑copy rates, and kernel launch queues. When a job stalls, they switch to the exporter (nvitop-exporter/cli.py) and feed the Prometheus metrics into Grafana, where they correlate GPU utilisation with training loss. The same metrics also trigger alerts if a process exceeds a configurable memory‑threshold, all without writing custom polling scripts.

Code Health & Issues

Measured static‑analysis findings (65 total)

[HIGH/soundness] Import‑cycle member x26 – files nvitop/api/__init__.py, nvitop/tui/library/__init__.py, nvitop/tui/tui.py participate in circular dependencies.
[HIGH/cognitive_load] Deep nesting x21 – max indentation depth 6 in nvitop/tui/tui.py, nvitop/api/device.py, nvitop/tui/library/keybinding.py.
[HIGH/cognitive_load] Oversized file x7 – nvitop/api/device.py 2688 lines, nvitop/api/utils.py, nvitop/api/process.py.
[HIGH/clarity] Duplicated code blocks – 292 repeated 6‑line fragments across 43 files (e.g., install-nvidia-driver.sh, nvitop/__init__.py, nvitop/api/__init__.py, nvitop/api/caching.py).
[MEDIUM/clarity] Hub module x3 – nvitop/api/__init__.py, nvitop/tui/library/__init__.py, nvitop/tui/tui.py are depended on by 17+ modules each.
[MEDIUM/resilience] Broad exception handling x2 – bare except in nvitop/tui/library/host.py and nvitop-exporter/nvitop_exporter/utils.py.

SDLC hygiene observations

[HIGH] Pin third‑party GitHub Actions to commit SHAs – current workflows use @release/v1, @v4 tags (.github/workflows/*).
[HIGH] Add a test suite – repository has no test files (57 source files, zero tests).
[HIGH] Commit a lockfile – nvitop-exporter/pyproject.toml lacks a generated lockfile.
[MEDIUM] Pin container base image by digest – Dockerfile uses ubuntu:latest.
[MEDIUM] Gate pull requests on a dependency vulnerability scan – no dependency-review-action in CI.
[MEDIUM] Set persist-credentials: false on checkout – build.yaml checkout retains the token.
[MEDIUM] Add a non‑root USER to the Docker image – no USER directive in any Dockerfile.

The Bottom Line

nvitop delivers a capable, single‑pane TUI and exporter for GPU‑process visibility, with solid API surface and Prometheus integration. However, the codebase suffers from import cycles, oversized files, duplicated logic, and missing reproducibility guards (no lockfile, mutable Docker base, unpinned Actions). Teams that need rapid GPU utilisation insights will find immediate value, but anyone planning to extend or ship the tooling should budget time to break the import cycles, split the large API files, and add a lockfile + CI dependency scanning before production use.