The Problem

Running large language models across multiple machines without a single point of failure is hard: there is no out‑of‑the‑box way to route requests, split layers, or keep owner‑control plane traffic separate from the public mesh. Users must stitch together GPU sharing, model loading, and API routing themselves, which quickly becomes a tangle of configuration and code.

What This Does

mesh‑llm is a collection of six self‑contained projects that together provide a distributed LLM runtime. The core crate crates/mesh-llm (crates/mesh-llm/src/commands/mod.rs:22) defines the CLI dispatch that reaches ~400 internal functions; it delegates to crates/mesh-llm-host-runtime for model inference and to crates/mesh-client for client‑side routing. Model layers are packaged as GGUF fragments in crates/mesh-llm-config (crates/mesh-llm-config/src/model/schema_types.rs) and fetched on‑demand. The public mesh is discovered and joined via crates/mesh-llm-commands (crates/mesh-llm-commands/src/setup/command_tests.rs:73), while owner‑control actions use an additive mesh-llm-control/1 lane bootstrap from crates/mesh-llm-config (crates/mesh-llm-config/src/authoring.rs). Inference flows through crates/mesh-llm-host-runtime/src/api/server.rs (handle_request) which calls a model for generation and makes outbound network calls when routing to a peer.

How It Is Wired

Execution starts at crates/llama-spec-bench/src/main.rs:184 (main) which reaches 399 functions but is called from nothing else in the repo. The CLI entry point handle in crates/mesh-client/src/runtime.rs:55 reaches 20 functions and is the first user‑facing call; it forwards to dispatch (crates/mesh-llm/src/commands/mod.rs:22). From there the call graph fans out: dispatch_command → dispatch_general_command → run_goose (command.status) launches a subprocess, while start → start_embedded_node (crates/mesh-llm-api-server/src/node.rs:308) loads a model via std::thread::Builder::new().name("mesh-llm-embedded"). The most‑connected internal symbols are is_empty (228 callers), Err (142), and map_err (133), indicating pervasive error‑handling patterns. Paths that leave the process are short: main → open_full_model (filesystem), dispatch → run_goose (subprocess), and start → start_embedded_node (model load). No circular dependencies exist among the 556 internal modules; the import graph is a clean DAG.

How To Use It

Setup – Install the executable (Linux/macOS):

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

Windows PowerShell:

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

Then initialise the local identity and config:

mesh-llm setup

Running a public mesh (exposes API on port 9337, web console on 3131):

mesh-llm serve --auto

For a private mesh with a specific model:

mesh-llm serve --model Qwen3-8B-Q4_K_M

To publish your own node so others can join:

mesh-llm serve --model Qwen3-8B-Q4_K_M --publish

Configuration lives in ~/.mesh-llm (created by setup). Environment variables such as MESH_LLM_MODEL can override the default; the config file is crates/mesh-llm-config/src/authoring.rs which defines the built‑in schema.

Real‑World Use

A developer wants to run a 7 B parameter model across two laptops. They start mesh-llm serve --auto --headless on each machine. The mesh discovers each other, negotiates layer ownership, and the coordinator (via crates/mesh-llm/src/commands/discover.rs) routes inference requests to the node that can serve the requested model. A curl call:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

returns a completion served locally or forwarded to the peer, transparently handling stage splits if the model exceeds a single GPU’s VRAM.

Code Health & Issues

Static analysis of 1392 code files reports 490 total findings (132 high, 343 medium, 15 low). High‑impact items include deep nesting in crates/mesh-client/src/client/builder.rs, crates/mesh-client/src/network/http_parse.rs, and crates/mesh-client/src/network/router.rs (max indentation depth 8); duplicated 6‑line blocks repeated 3838 times across 265 files; and oversized files crates/mesh-llm-cli/src/parser.rs, crates/mesh-llm-config/src/model/built_in_schema.rs, and crates/mesh-llm-config/src/validate.rs (1459 lines each).

Beyond the measured analysis, the SDLC audit (8 findings) flags the following production‑reach issues:

  • High – Pin third‑party GitHub Actions to a commit SHA in .github/workflows (currently dorny/paths-filter@v4, pnpm/action-setup@v4, dtolnay/rust-toolchain@stable, mozilla-actions/sccache-action@v0.0.9); a tag can move and run actions with your secrets under new ownership.
  • Medium – Enable Dependabot or Renovate; 61 manifests have no update bot, leaving known advisories unpatched.
  • Medium – Pin the container base image by digest in docker/Dockerfile.client (currently node:24-alpine, debian:bookworm-slim); mutable tags mean different libc and CVEs between builds.
  • Medium – Gate pull requests on a dependency vulnerability scan; no dependency‑scan step exists in CI.
  • Medium – Set persist-credentials: false on checkout (docker-precheck.yml); the token remains in .git/config for later steps.
  • Medium – Add a non‑root USER to the image (docker/Dockerfile.client); currently the entrypoint runs as root, expanding any container escape to host‑level impact.
  • Low – Set timeout-minutes on workflow jobs in docker-precheck.yml; three jobs have no timeout, risking overlap on a two‑hourly schedule.
  • Low – Add convention files (.editorconfig, .gitattributes with text=auto eol=lf, formatter config); their absence can cause mixed line endings and unreviewable diffs.

The Bottom Line

mesh‑llm delivers a usable distributed LLM runtime with a clear OpenAI‑compatible API, model‑layer packaging, and mesh‑wide routing, all driven by a modest Rust/TypeScript codebase. The architecture is well‑structured (no circular deps, clean import DAG) and the quick‑start commands are concrete. However, the codebase suffers from deep nesting, duplicated logic, and oversized files that hinder maintenance; the hygiene audit reveals actionable SDLC gaps (action pinning, Dependabot, image digest pinning, dependency scanning, non‑root containers). Teams that need to run private or public LLM meshes will find value here, provided they invest in the recommended pinning and bot setup to keep the environment secure and maintainable.