The Problem
Researchers building large‑scale agentic reinforcement learning systems struggle with a fragmented stack: separate libraries for distributed training, inference, and model storage. Each component usually requires a custom integration layer, leading to duplicated effort and fragile pipelines.
What This Does
molt provides a unified, Hugging‑Face‑native framework that bundles Ray for distributed execution, vLLM for fast roll‑outs, and NeMo AutoModel + FSDP2 for efficient actor training. The code base is under 9 k LOC, keeping the learning curve shallow while still supporting 1 T‑class MoE models. Key entry points are molt/cli/train_rl_ray.py for reinforcement learning and molt/cli/train_sft.py for supervised fine‑tuning. The molt/models package exposes actor.py, critic.py, and loss.py, which the trainers import directly. Dataset utilities reside in molt/datasets/sft_dataset.py and molt/datasets/prompts_dataset.py. Utilities for distributed batching live in molt/utils/distributed_sampler.py. The Docker image, built from dockerfile/Dockerfile, bundles all runtime dependencies, enabling a single container to run both training and inference.
How It Is Wired
Entry point: python -m molt.cli.train_rl_ray
train_rl_ray.pyparses CLI args (viamolt/cli/common_args.py) and loads a config object.- It constructs a
RLTrainer(molt/trainer/rl_trainer.py) which internally creates aRolloutgenerator usingmolt/trainer/rollout/experience_maker.py. - The rollout generator calls
vllm/vllm_engine.pyto produce token streams; the resulting samples are passed to the actor loss (molt/models/loss.py). - The actor is wrapped by FSDP (
molt/trainer/fsdp/muon.py) and Adam CPU‑offload (optimizer_offload.py). - Checkpointing occurs in
molt/trainer/fsdp/checkpoint.py, which writes to a single directory that vLLM can read back for inference.
The most central hub is RLTrainer; it orchestrates data flow between the Ray worker pool, vLLM workers, and the FSDP‑wrapped actor. Modifying any part of this pipeline requires editing at least one of the following files: train_rl_ray.py, rl_trainer.py, experience_maker.py, vllm_engine.py, or checkpoint.py. The dependency graph shows a single cycle between rl_trainer.py and vllm_engine.py (via the Rollout object), which is the only place where the training and inference loops touch each other.
How To Use It
Setup
# From the repo root
pip install -e . # installs the `molt` package
# or build a container
docker build -t molt:dev -f dockerfile/Dockerfile .
Configuration molt/utils/config.py loads a YAML config; a minimal example lives in examples/scripts/quick_start/rl_qwen3_4b.sh. No explicit environment variables are required beyond standard Hugging Face tokens (HF_TOKEN).
Running
# From the repo root, using the quick‑start script
bash examples/scripts/quick_start/rl_qwen3_4b.sh
# Or manually
python -m molt.cli.train_rl_ray --config examples/configs/rl_qwen3.yaml
The script launches a Ray cluster, spins up vLLM workers, and starts training a 4 B model.
Real‑World Use
A research lab can spin up a single node with 8 GPU, run the provided rl_flash_reinforce_r1d_1p5b.sh script, and obtain a 20 000‑step REINFORCE‑zero policy with a single rollout per step. The output checkpoint is immediately usable by a production inference server built on vLLM.
Code Health & Issues
- Low – Missing lockfile:
docs/fern/package.jsondeclares npm dependencies but nopackage-lock.json. - Low – Untracked tests:
tests/e2econtains scripts but no automated test target in GitHub Actions; CI may miss end‑to‑end regressions. - Low – Documentation gaps:
docs/guide/*.mdlack examples for custom environment creation; developers must consult the code directly.
No critical security or license issues were detected.
The Bottom Line
molt delivers a compact, well‑structured framework that removes the boilerplate of integrating Ray, vLLM, and NeMo. Its modular architecture keeps the core train loop in a single file (rl_trainer.py), making upstream changes straightforward. The main risk is the absence of a lockfile for the front‑end assets, which can affect reproducibility. Researchers or engineers comfortable with Ray and Hugging Face will find molt a pragmatic baseline for large‑scale agentic RL experiments.