The Problem

Teams that need to fine‑tune, evaluate, or ship open‑source LLMs/VLMs (e.g., GPT‑OSS, Qwen‑3, DeepSeek‑R1) must stitch together data pipelines, trainer scripts, and deployment wrappers. Doing this manually incurs duplicated boiler‑plate, fragile config handling, and ad‑hoc Docker builds, which slows iteration and raises reproducibility risk.

What This Does

oumi bundles four self‑contained projects that cover the full model‑life‑cycle:

  • Fine‑tuning & hyper‑parameter search – scripts under src/oumi/core/tune/ and corresponding YAML recipes in configs/recipes/.
  • Evaluation & benchmarking – src/oumi/analyze/ provides analyzers, while configs/apis/*/eval_*.yaml encode provider‑specific evaluation calls.
  • Inference & serving – the CLI (src/oumi/cli/main.py) launches local or remote inference engines defined in src/oumi/launcher/.
  • Data synthesis & preprocessing – notebooks in notebooks/ and helper scripts in scripts/datasets/ generate synthetic corpora.

Configuration lives in the massive configs/ tree (540 YAML files) and is consumed by the CLI via src/oumi/config/loader.py. The package is container‑ready (Dockerfile), pip‑installable (pyproject.toml), and CI‑tested (GitHub Actions in .github/workflows/).

How It Is Wired

Execution starts at src/oumi/__main__.py, which simply calls oumi.cli.main():

# src/oumi/__main__.py
if __name__ == "__main__":
    from oumi.cli.main import main
    main()

src/oumi/cli/main.py builds an argparse CLI, loads a YAML recipe (e.g., configs/recipes/gpt_oss/train.yaml), and dispatches to one of three sub‑commands:

  • tune → src/oumi/core/tune/engine.py → launches a training loop that pulls data via src/oumi/dataset/loader.py and writes checkpoints using src/oumi/launcher/clients/local_client.py.
  • eval → src/oumi/analyze/evaluator.py → iterates over a test set, calling provider‑specific inference functions in src/oumi/launcher/clients/.
  • infer → src/oumi/launcher/clients/remote_client.py or local_client.py → opens sockets, streams tensors, and writes results to output/.

Only 26 import edges connect the 569 internal modules, and there are no circular imports, indicating a deliberately shallow dependency graph. The most‑connected non‑test module is tests/markers.py, which is imported by 15 test files, making it a potential blast‑radius hotspot. All side‑effects (file writes, network calls) are confined to the launcher package; core analytics and tuning logic are pure‑Python functions that return data structures.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/oumi
cd oumi

# Build the container (optional)
docker build -t oumi:latest .

# Install the Python package (uses uv if available)
uv pip install -e .

# Run a fine‑tuning job (example recipe)
oumi tune --config configs/recipes/gpt_oss/train.yaml

Configuration files required by the CLI are the YAML recipes under configs/recipes/. Provider API keys must be exported as environment variables referenced in those YAMLs (e.g., OPENAI_API_KEY). No additional setup scripts are present; the Makefile offers make test and make lint targets that invoke the GitHub Actions workflows locally.

Real‑World Use

A data‑science team can spin up a GPU‑enabled container, point the CLI at a configs/recipes/qwen3/train.yaml recipe, and obtain a checkpoint ready for deployment with a single oumi infer --model checkpoint.pt command. The same CLI can later run oumi eval --config configs/recipes/qwen3/eval.yaml to generate a PDF report via src/oumi/analyze/report_generator.py.

Code Health & Issues

  • High – Pin GitHub Actions – .github/workflows/*.yaml use tags (@v7, @v3). Replace with 40‑character commit SHAs.
  • High – Add lockfile – pyproject.toml lacks a lockfile; commit uv.lock (or poetry.lock).
  • Medium – Pin Docker base images – Dockerfile references mutable tags (pytorch/pytorch:${PYTORCH_VERSION}-cuda${CUDA_VERSION}); use digests.
  • Medium – Add dependency‑vulnerability scan – No scan step in CI; integrate dependency-review-action or osv-scanner.
  • Medium – Disable credential persistence – .github/workflows/doctests.yaml checks out with token persistence; add persist-credentials: false.
  • Low – Set workflow timeouts – Jobs in doctests.yaml lack timeout-minutes; specify reasonable limits.

Additional measured findings: duplicated 6‑line blocks across > 480 files, deep nesting (max depth 9) in several scripts, three test files > 1300 lines, file opens without context managers in three core modules, and broad except: clauses in three analysis modules. The hub module tests/markers.py is imported by 15 tests, so changes there ripple widely.

The Bottom Line

oumi delivers a practical, end‑to‑end toolbox for open‑source LLM/VLM workflows, with clear entry points and a modular import graph. Reproducibility is weakened by missing lockfiles and mutable Docker tags, and code hygiene suffers from duplicated snippets and deep nesting. Teams comfortable with Python and Docker will find it immediately useful; they should address the high‑severity health items before adopting it in production.