LLMRouter - Moses Yebei

The Problem

Organizations running LLM inference face a cost-quality tradeoff: small models are cheap but underperform on complex queries, while large models are expensive and slow for trivial ones. LLMRouter solves this by routing each query to the most appropriate model dynamically, based on task complexity, cost constraints, and performance requirements—reducing inference spend without sacrificing output quality.

What This Does

LLMRouter is a Python library implementing 16+ routing strategies across five categories: single-round, multi-round, multimodal, agentic, and personalized routers. The llmrouter/ directory (141 files) contains the core implementations, including KNN, SVM, MLP, matrix factorization, Elo rating, and graph-based approaches. Each router is defined by a YAML config in configs/model_config_train/ or configs/model_config_test/.

The repo includes a complete data generation pipeline (benchmark_pipeline/) that pulls from 11 benchmark datasets, generates embeddings, and runs automated evaluation. A unified CLI supports training, inference, and interactive chat via Gradio. The xRouteBench dataset (hosted on Hugging Face) provides standardized evaluation data.

How It Is Wired

The primary entry point is llmrouter/serve/server.py, which starts a model-serving HTTP endpoint. The training path flows through llmrouter/utils/setup.py and config files in configs/, with each router class loading its hyperparameters from YAML. The openclaw_router/ module (9 files) provides an alternative server implementation with __main__.py as its entry point.

The benchmark_pipeline/ runs as a discrete workflow: run_pipeline.py orchestrates download_data.py → generate_embeddings.py → _causallm_stage.py → aggregate_results.py. This pipeline touches the filesystem heavily (downloading datasets, writing embeddings) and calls external APIs for model inference.

The ComfyUI/ integration (2 files) exposes routing as a ComfyUI node, letting users connect LLMRouter into visual workflow graphs.

The module graph shows a hub around llmrouter/ core, with most routers importing shared utilities. The custom_routers/ directory (9 files) is the extension point—new routing strategies plug in here without touching core code.

How To Use It

Setup — pyproject.toml is present, so install with pip or uv. No lockfile exists, so builds are not fully reproducible.

git clone https://github.com/moses-y/LLMRouter
cd LLMRouter
pip install -e .

Configuration — Model and router settings live in configs/model_config_train/*.yaml and configs/model_config_test/*.yaml. The benchmark_pipeline/configs/hparams.yaml controls benchmark hyperparameters. API keys for model providers go in environment variables (not documented in the repo).

Running it — The README documents a CLI interface but does not show verbatim commands. The server starts via llmrouter/serve/server.py; the benchmark runs via benchmark_pipeline/run_pipeline.py. Interactive chat uses a Gradio UI launched from the CLI.

Real-World Use

A production deployment would run LLMRouter as a proxy in front of multiple model endpoints. Each incoming query is scored by a trained router (e.g., KNN or MLP), which selects the cheapest model predicted to meet quality thresholds. The router learns from historical query-response pairs, improving routing decisions over time. The personalizedrouter.yaml config extends this to per-user routing, useful for SaaS products with heterogeneous user bases.

Code Health & Issues

Low - No dependency lockfile — pyproject.toml declares dependencies without pinned versions, so builds are non-reproducible across environments.
Low - No CI pipeline — .github/workflows/ contains only deploy-docs.yml; there is no automated test execution despite 55 test files existing.
Low - Stale fork — This is a fork of ulab-uiuc/LLMRouter with 0 stars; verify it tracks upstream before relying on it.

The Bottom Line

LLMRouter is a serious, research-grade routing library with genuine breadth—16+ strategies, a full benchmark pipeline, and a clean extension point in custom_routers/. The lack of a lockfile and CI is acceptable for a research tool but needs addressing for production use. Teams already using multiple LLM endpoints and wanting to cut costs without quality loss should evaluate this; teams needing a battle-tested, dependency-pinned proxy should look elsewhere.

What the analyser found

Deployment readiness

5/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

349 files

Python181
YAML54
Markdown33
Jupyter Notebook31
JSON7
Shell6

PyTorchpandas

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["llmrouter 105 modules"] g1["tests 32 modules"] g2["data 20 modules"] g3["openclaw_router 7 modules"] g4["custom_routers 7 modules"] g5["benchmark_pipeline 5 modules"] g6["custom_tasks 3 modules"] g1 -->|31| g0 g6 -->|7| g0 g5 -->|6| g0 g2 -->|6| g0 g4 -->|4| g0 g0 -->|2| g4 g1 -->|2| g3 g0 -->|1| g3 g3 -->|1| g0 g1 -->|1| g4

Module dependencies

The 10 most depended-upon modules of 181, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_llmrouter_utils___init__["__init__"] n1_llmrouter_models___init__["__init__"] n2_llmrouter_models_meta_router["meta_router"] n3_lmrouter_models_base_trainer["base_trainer"] n4_llmrouter_data___init__["__init__"] n5_llmrouter_prompts___init__["__init__"] n6_openclaw_router_config["config"] n7_llmrouter_utils_evaluation["evaluation"] n8_llmrouter_utils_prompting["prompting"] n9_llmrouter_data_data_loader["data_loader"] n4_llmrouter_data___init__ --> n9_llmrouter_data_data_loader n9_llmrouter_data_data_loader --> n0_llmrouter_utils___init__ n1_llmrouter_models___init__ --> n2_llmrouter_models_meta_router n1_llmrouter_models___init__ --> n3_lmrouter_models_base_trainer n2_llmrouter_models_meta_router --> n4_llmrouter_data___init__ n0_llmrouter_utils___init__ --> n8_llmrouter_utils_prompting n0_llmrouter_utils___init__ --> n7_llmrouter_utils_evaluation n7_llmrouter_utils_evaluation --> n8_llmrouter_utils_prompting n8_llmrouter_utils_prompting --> n5_llmrouter_prompts___init__ classDef cyc stroke:#ef4444,stroke-width:2px; class n0_llmrouter_utils___init__,n4_llmrouter_data___init__,n7_llmrouter_utils_evaluation cyc;

Dependencies

6 of 20 declared dependencies are a major version behind. Resolved 20 of 20 against the registry.

Package	Declared	Latest	Behind
`protobuf`	>=3.20	7.35.1	4 major
`datasets`	>=2.14	5.0.1	3 major
`pandas`	>=1.5	3.0.5	2 major
`gradio`	>=4.0	6.24.0	2 major
`transformers`	>=4.40	5.15.0	1 major
`numpy`	>=1.21	2.5.2	1 major

Findings

LowDependencies declared without a lockfile - non-reproducible builds pyproject.toml

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium5

low1

Commit a lockfile beside the manifesthigh

pyproject.toml

manifest with no lockfile in scope

An unlocked range means the artifact you tested and the artifact you ship can contain different transitive code, so a malicious patch release reaches production with no diff.

Fix: Run the package manager once and commit the generated lockfile.

Make CI invoke the test suite it hashigh34 occurrences

.github/workflows

34 test files, no test command in any workflow

A green check that never executed an assertion is worse than no check, because reviewers trust it.

Fix: Add a test step to the existing workflow rather than a new workflow.

Enable Dependabot or Renovatemedium

1 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Split the oversized notebook cell into stepsmedium

notebooks/data_preparation/01_data_preparation.ipynb

largest cell 5068 characters of 18860 (3 notebooks sampled)

A failure anywhere in a 22,000-character cell means rerunning the whole pipeline from load to model, so nothing can be debugged in isolation.

Fix: Break at the load, clean, feature, model and evaluate boundaries.

Move large binaries to Git LFS or out of the repositorymedium9 occurrences

data/default_data/default_routing_train_data.jsonl

9 blobs over 5MB: default_routing_train_data.jsonl 57.9MB, default_routing_train_data.jsonl 57.9MB, default_routing_train_data.csv 46.0MB

One repository here carries twenty blobs over 5MB including a 10.9MB spreadsheet, so every clone and every CI checkout pays for data nobody diffs.

Fix: Track those extensions with LFS, or move datasets to object storage and fetch them in a setup step.

Set persist-credentials: false on checkoutmedium

.github/workflows/deploy-docs.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Set timeout-minutes on the workflow jobslow

.github/workflows/deploy-docs.yml

1 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.