The Problem

When a single LLM‑driven “Oh My Pi” (OMP) agent fails, developers must manually iterate to find a fix. Running several candidate agents in parallel raises the chance of a correct solution, but selecting the best trajectory and applying its patch safely is non‑trivial, especially in a clean Git state.

What This Does

omp-best-of orchestrates N OMP agents in isolated copy‑on‑write workspaces, captures each full transcript and generated patch, and ranks the results with an LLM‑as‑a‑Verifier or a sampled pairwise judge. The winner can be applied to the original HEAD with --apply.

Key locations:

  • CLI – src/cli.ts parses flags (--n, --apply, --verifier-thinking) and launches the runner.
  • Runner – src/runner.ts creates the baseline worktree, fans out candidate sessions via src/extension.ts, and collects artifacts.
  • Verifier – src/verifier.ts (continuous‑logprob) and src/sampled-verifier.ts (pairwise) score trajectories.
  • Benchmarks – bench/ holds four independent task suites (e.g., async-memoize, circuit-breaker) each with its own repo/package.json, reference implementation, oracle tests, and result JSON files.

The repository is a portfolio of four self‑contained benchmark projects plus the core OMP orchestration (src/) and supporting scripts (scripts/, python/).

How It Is Wired

  1. Entry point – src/cli.ts invokes runBestOf (exported from src/runner.ts).
  2. Preflight – src/runner.ts calls ensureCleanTree() (checks git status) and records HEAD.
  3. Workspace creation – src/extension.ts uses OMP’s omp.workspaces.create() to spin up isolated copy‑on‑write workspaces; fallback is a Git worktree (see bench/run.ts for the low‑level loop).
  4. Candidate execution – each workspace runs omp run --json (headless OMP session) and streams JSON events to src/process.ts, which extracts exit code, usage, and raw transcript.
  5. Patch capture – src/model.ts computes a binary‑safe diff via OMP’s baseline‑aware delta logic.
  6. Verification – src/verifier.ts (logprob) or src/sampled-verifier.ts (sampled) receives the trajectory objects, calls the external verifier endpoint (default DeepSeek V4 Flash) and returns a score.
  7. Selection & Apply – src/runner.ts picks the highest‑scoring candidate; if --apply is set and the repository is still clean, it runs git apply with the stored patch.
  8. Cleanup – all workspaces are torn down, artifacts are written under bench/results/.

The internal call graph shows src/runner as a hub (5 inbound, 6 outbound imports, instability 0.55). No circular dependencies were detected, simplifying future refactors. Deep nesting (max depth 6) and high branching density (e.g., 51 branches in src/args.ts) increase cognitive load, and duplicated 6‑line blocks appear across multiple files (bench/run, src/args, etc.).

How To Use It

# Clone the repo
git clone https://github.com/moses-y/omp-best-of
cd omp-best-of

# Install Bun (required runtime) and dependencies
bun install   # reads the many bench/*/repo/package.json files

# Verify a clean working tree, then run 5 candidates and apply the best patch
bun src/cli.ts --n 5 --apply "Fix the failing authentication test"

No .env file is required; the verifier endpoint is resolved from OMP’s model registry. To switch verifier backends, add --verifier logprob|sampled (default logprob). The benchmark suites can be exercised directly, e.g.:

bun bench/run.ts async-memoize

Real‑World Use

A CI job that runs nightly can invoke src/cli.ts with --n 3 on a failing test suite. The selected patch is automatically committed if the repository remains unchanged, providing a “self‑healing” feedback loop without human intervention.

Code Health & Issues

  • High – GitHub Actions not pinned: .github/workflows/ci.yml uses oven-sh/setup-bun@v2. Pin to a commit SHA.
  • Medium – No dependency‑vulnerability gate: add dependency-review-action or osv-scanner to the CI workflow.
  • Low – Missing repository conventions: add .editorconfig, .gitattributes, and a formatter config to enforce consistent style.
  • Low/Risk – No lockfile for bench task packages (e.g., bench/tasks/async-memoize/repo/package.json), leading to non‑reproducible installs.
  • Tests exist (57 files) and CI runs on push; license (MIT) is present; no secrets detected.

The Bottom Line

omp-best-of delivers a concrete pipeline for parallel OMP candidate generation and verifier‑driven selection, with a clear CLI and isolated workspace handling. The codebase is functional but suffers from deep nesting, branching density, and duplicated snippets that make maintenance harder. Pinning CI actions, adding a vulnerability scan, and introducing lockfiles would raise production readiness. Suitable for teams already using OMP who need automated “best‑of‑N” patch generation, but expect to invest in refactoring for long‑term stability.