verifiers - Moses Yebei

The Problem

Training and evaluating large‑language‑model (LLM) agents requires a repeatable harness: a dataset, a tool‑enabled runtime, and a scoring rubric. Maintaining these pieces across many tasks quickly becomes ad‑hoc, leading to duplicated scripts, fragile imports, and opaque build artefacts.

What This Does

verifiers is a modular library that packages environments, rubrics, and utility helpers for LLM reinforcement‑learning. The repository is a portfolio of five self‑contained projects:

Project	Core purpose	Key files
`verifiers/`	Core SDK – types, environment base classes, rubric API	`verifiers/__init__.py`, `verifiers/types.py`, `verifiers/envs/environment.py`, `verifiers/rubrics/rubric.py`
`environments/`	Ready‑made RL tasks (e.g., `alphabet_sort`, `browser_cua_example`)	`environments/*/pyproject.toml`, `environments/openenv_echo/proj/server/app.py`
`packages/`	Optional extensions (e.g., `verifiers-rl`)	`packages/verifiers-rl/verifiers_rl/scripts/rl.py`
`scripts/`	CLI utilities for inspection and TUI	`verifiers/scripts/tui.py`
`configs/`	Global configuration scaffolding	`configs/endpoints.py`

Each environment ships a small Flask server (see environments/openenv_echo/proj/server/app.py) that loads the dataset, instantiates a verifiers.envs.Environment subclass, and registers a verifiers.rubrics.Rubric for reward calculation.

How It Is Wired

Execution begins at the Flask entry point in environments/openenv_echo/proj/server/app.py (function main). The call graph shows:

main → load_environment (creates the concrete Environment subclass).
load_environment → Rubric (adds reward functions, see verifiers/rubrics/rubric.py).
Rubric is the most‑used hub (imported by 79 modules) and subsequently invoked by the rollout loop in verifiers/envs/environment.py (rollout → make_input → Parser).

The most blast‑radius functions are Rubric.__init__ (112 callers) and make_input (79 callers). They sit in the hub module verifiers/__init__.py, which also participates in a circular import chain with verifiers/utils/save_utils.py and verifiers/rubrics/rubric.py. The cycle inflates the instability metric (0.28 for __init__, 0.54 for save_utils) and makes refactoring risky.

File‑system interaction occurs chiefly in verifiers/scripts/tui.py (metadata loading, result discovery) and verifiers/envs/experimental/rlm_env.py (tool merging, external command execution). Network calls are limited to the Flask servers and a few outbound client.post requests in the rollout path.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/verifiers
cd verifiers

# Install Python dependencies (uv is the recommended manager)
uv sync          # reads the pyproject.toml files in envs/* and verifiers/

# Build the browser‑CUA template (optional UI component)
cd assets/templates/browserbase/cua
docker build -f Dockerfile.build -t verifiers-cua:latest .

To run a sample environment locally:

# Open a Flask server for the echo environment
cd environments/openenv_echo/proj/server
uv run app.py   # starts the server on http://localhost:5000

The server will automatically load the environment, instantiate its rubric, and expose /rollout endpoints used by the RL trainer (packages/verifiers-rl/...).

Real‑World Use

A downstream RL pipeline imports verifiers.envs.Environment to create a training loop:

from verifiers.envs import SimpleEnvironment
from verifiers.rubrics import Rubric

env = SimpleEnvironment(dataset="my_dataset")
rubric = Rubric()
env.attach_rubric(rubric)

for episode in range(1000):
    traj = env.rollout(agent)
    reward = rubric.score(traj)
    trainer.update(traj, reward)

The SimpleEnvironment class lives in verifiers/envs/environment.py; the rubric scoring path goes through verifiers/rubrics/rubric.py, the same hub that other environments share, guaranteeing consistent metric definitions across projects.

Code Health & Issues

Measured findings (static analysis)

Import cycles (high) – 25 files, e.g., verifiers/__init__.py, verifiers/utils/save_utils.py, verifiers/rubrics/rubric.py. Break cycles by extracting shared types or deferring imports.
Deep nesting (high) – up to 6‑level indentation in verifiers/types.py and verifiers/envs/environment.py. Refactor with guard clauses.
Hub module (high) – verifiers/__init__.py is imported by 79 modules; keep it minimal.
Oversized files (high) – verifiers/envs/environment.py (>1 200 LOC). Split by responsibility.

Health audit (repo‑level)

HIGH Pin GitHub Actions to commit SHAs – .github/workflows/*.
HIGH Add lockfile for assets/templates/browserbase/cua/package.json.
MEDIUM Declare least‑privilege GITHUB_TOKEN permissions – publish-envs.yml.
MEDIUM Enable Dependabot / Renovate for 28 manifests.
MEDIUM Pin Docker base image by digest – Dockerfile.build.
MEDIUM Add dependency‑vulnerability scan step in CI.
MEDIUM Set persist-credentials: false on checkout – test.yml.
MEDIUM Add non‑root USER to Dockerfile.
LOW Set timeout-minutes on workflow jobs – publish-envs.yml.

Other hygiene: tests and CI are present, a license file exists, but no lockfile for the Node template and no explicit least‑privilege token settings.

The Bottom Line

verifiers offers a concrete, reusable stack for LLM RL environments, with clear separation of datasets, runtimes, and rubrics. The codebase is functional but suffers from circular imports, oversized core modules, and several CI‑security gaps that should be addressed before production deployment. Teams comfortable with Python/Flask and container builds will find the library extensible; fixing the highlighted health issues will markedly improve maintainability and supply‑chain safety.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

386 files

Python197
Markdown89
TOML52
YAML11
Shell11
JSON7

ReactFlaskDocker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["verifiers 98 modules"] g1["tests 46 modules"] g2["environments 37 modules"] g3["packages 12 modules"] g4["assets 6 modules"] g5["scripts 3 modules"] g1 -->|83| g0 g2 -->|40| g0 g3 -->|8| g0 g5 -->|2| g0 g1 -->|2| g2

Module dependencies

The 10 most depended-upon modules of 203, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_verifiers___init__["__init__"] n1_verifiers_types["types"] n2_erifiers_utils_message_utils["message_utils"] n3_verifiers_utils_save_utils["save_utils"] n4_verifiers_parsers_parser["parser"] n5_verifiers_utils_data_utils["data_utils"] n6_verifiers_rubrics_rubric["rubric"] n7_verifiers_utils_eval_utils["eval_utils"] n8_verifiers_utils_async_utils["async_utils"] n9_verifiers_utils_path_utils["path_utils"] n0_verifiers___init__ --> n1_verifiers_types n0_verifiers___init__ --> n4_verifiers_parsers_parser n0_verifiers___init__ --> n6_verifiers_rubrics_rubric n0_verifiers___init__ --> n5_verifiers_utils_data_utils n4_verifiers_parsers_parser --> n1_verifiers_types n6_verifiers_rubrics_rubric --> n0_verifiers___init__ n6_verifiers_rubrics_rubric --> n1_verifiers_types n6_verifiers_rubrics_rubric --> n8_verifiers_utils_async_utils n8_verifiers_utils_async_utils --> n0_verifiers___init__ n5_verifiers_utils_data_utils --> n1_verifiers_types n7_verifiers_utils_eval_utils --> n0_verifiers___init__ n7_verifiers_utils_eval_utils --> n1_verifiers_types n7_verifiers_utils_eval_utils --> n8_verifiers_utils_async_utils n7_verifiers_utils_eval_utils --> n9_verifiers_utils_path_utils n2_erifiers_utils_message_utils --> n1_verifiers_types n9_verifiers_utils_path_utils --> n1_verifiers_types n3_verifiers_utils_save_utils --> n1_verifiers_types n3_verifiers_utils_save_utils --> n2_erifiers_utils_message_utils classDef cyc stroke:#ef4444,stroke-width:2px; class n0_verifiers___init__,n3_verifiers_utils_save_utils,n6_verifiers_rubrics_rubric,n7_verifiers_utils_eval_utils,n8_verifiers_utils_async_utils cyc;

Findings

LowDependencies declared without a lockfile - non-reproducible builds assets/templates/browserbase/cua/package.json

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium6

low1

Pin third-party GitHub Actions to a commit SHAhigh6 occurrences

.github/workflows

tj-actions/changed-files@v47, astral-sh/setup-uv@v5, astral-sh/setup-uv@v6, astral-sh/setup-uv@v3

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Commit a lockfile beside the manifesthigh

assets/templates/browserbase/cua/package.json

manifest with no lockfile in scope

An unlocked range means the artifact you tested and the artifact you ship can contain different transitive code, so a malicious patch release reaches production with no diff.

Fix: Run the package manager once and commit the generated lockfile.

Declare least-privilege permissions for GITHUB_TOKENmedium3 occurrences

.github/workflows/publish-envs.yml

3 workflow(s) declare no permissions, 3 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Enable Dependabot or Renovatemedium

28 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

assets/templates/browserbase/cua/Dockerfile.build

node:22-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Set persist-credentials: false on checkoutmedium

.github/workflows/test.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

assets/templates/browserbase/cua/Dockerfile.build

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/publish-envs.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.