MiniCPM o - Moses Yebei

The Problem

Developers need an on‑device multimodal LLM that can ingest image, video, audio and text while producing simultaneous speech and text output. Existing open‑source MLLMs either lack full‑duplex streaming or require heavyweight servers, making real‑time phone deployment impractical.

What This Does

MiniCPM‑o ships four self‑contained projects:

eval_mm – evaluation toolkit (eval_mm/vlmevalkit/) with dataset loaders, metric helpers and a CLI (eval_mm/vlmevalkit/run.py).
web_demos – a browser‑based demo (web_demos/minicpm‑o_2.6/web_server/) that runs the model through a WebRTC‑enabled UI.
omnilmm – core model utilities (omnilmm/model/, omnilmm/conversation.py) that expose inference APIs.
finetune – scripts for further model fine‑tuning (finetune/).

Common code lives in Python (118 files) and the front‑end uses Vue/React with pnpm as the package manager. Docker is used for the demo server.

How It Is Wired

Execution starts at eval_mm/vlmevalkit/run.py:main. The main function (reaches 317 functions) parses CLI args, builds a prompt via eval_mm/vlmevalkit/vlmeval/smp/file.py::build_prompt, loads a dataset (eval_mm/vlmevalkit/vlmeval/dataset/...), and calls evaluate.

Key call chain (high blast‑radius nodes are bold):

main → evaluate → load (x117) → build_prompt (x114) → dump (x92) → track_progress_rich (x24)

load reads files from disk; build_prompt constructs model input; dump writes results; track_progress_rich prints progress via Rich.

evaluate also invokes model.generate (28 calls) which lives in omnilmm/model/utils.py. This is the only place that contacts the inference backend (CPU/GPU or ONNX runtime).

Web demo starts with web_demos/minicpm‑o_2.6/web_server/src/apis/index.js, which imports omnilmm/model/utils.py through a Python‑backed HTTP endpoint defined in web_demos/.../index.html. The Dockerfile builds a container that installs pnpm dependencies (pnpm install) and the Python environment (pip install -r requirements.txt).

Circular import warning: eval_mm/vlmevalkit/vlmeval/dataset/__init__.py participates in a 7‑module cycle, raising the risk of import‑time side effects when refactoring.

Hub modules – eval_mm/vlmevalkit/vlmeval/smp/__init__.py is imported by 57 other modules; changes here propagate widely.

External effects – 50 functions perform filesystem I/O, 3 launch subprocesses, 28 call the model for inference, and 1 performs a cryptographic operation. The only outbound network call is in eval_mm/vlmevalkit/vlmeval/api/base.py when contacting a remote service for content checking.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/MiniCPM-o
cd MiniCPM-o

# Install Python deps (base environment)
pip install -r requirements.txt   # includes eval_mm, finetune, omnilmm

# Run an evaluation (example VQA)
python -m eval_mm.vlmevalkit.run \
    --dataset vqa --model MiniCPM-o-4.5 \
    --output results.json

To launch the web demo locally:

# Install front‑end deps
cd web_demos/minicpm-o_2.6/web_server
pnpm install

# Build and run the Docker image
docker build -t minicpm-o-demo .
docker run -p 8080:80 minicpm-o-demo

Configuration – The demo and evaluation scripts read environment variables from .env files (eval_mm/vlmevalkit/.env, web_demos/.../.env.*). Replace placeholder keys (e.g., OPENAI_API_KEY) with your own credentials or remove the files after creating .env.example.

Real‑World Use

A mobile app can embed the ONNX‑exported MiniCPM‑o model, stream video/audio to a backend running omnilmm/model/utils.py, and receive simultaneous transcription and response audio. The evaluation toolkit can be used to benchmark the on‑device model against VQA, OCR, and speech benchmarks before deployment.

Code Health & Issues

Critical – Rotate and remove committed credentials in eval_mm/vlmevalkit/.env.

High – No test suite; add tests for each public entry point. High – No CI workflow; add GitHub Actions to build and test. High – Deployable Dockerfile lacks a build gate; add a CI step that builds the image. High – Remove committed .env files and replace with .env.example.

Medium – Enable Dependabot for Python, npm, and Docker manifests. Medium – Pin Docker base images by digest. Medium – Add a pre‑commit secret‑scan hook. Medium – Move large binary assets (e.g., assets/minicpmv4_5/MiniCPM-V 4.5-8.26.mp4) to Git LFS or external storage.

Low – Add repository convention files (.editorconfig, .gitattributes, formatter config).

Additional static findings: 25 files with deep nesting (max indentation 6), 7 import cycles, 6 “open without context manager”, 13 broad exception handlers, duplicated 6‑line code blocks across 68 files, and two oversized modules (>1 400 lines). These are documented in the measured analysis and should be prioritized for refactoring.

The Bottom Line

MiniCPM‑o provides a functional on‑device multimodal LLM with full‑duplex streaming, backed by a concrete evaluation suite and a ready‑to‑run web demo. The codebase is functional but suffers from missing tests, CI, and several maintainability hotspots (import cycles, large modules, duplicated code). It is suitable for research or prototype integration, but production teams should first address the critical credential leak and add automated testing and CI pipelines.

What the analyser found

Deployment readiness

5/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

389 files

Python118
Markdown33
Vue20
JavaScript18
JSON8
YAML7

ReactVueDockerONNX

Module dependencies

The 10 most depended-upon modules of 138, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_evalkit_vlmeval_smp___init__["__init__"] n1_alkit_vlmeval_utils___init__["__init__"] n2_t_vlmeval_dataset_image_base["image_base"] n3_meval_dataset_utils___init__["__init__"] n4_ataset_utils_multiple_choice["multiple_choice"] n5_t_vlmeval_dataset_video_base["video_base"] n6__mm_vlmevalkit_vlmeval_tools["tools"] n7_kit_vlmeval_dataset___init__["__init__"] n8_val_dataset_utils_judge_util["judge_util"] n9_utils_ccocr_evaluator_common["common"] n7_kit_vlmeval_dataset___init__ --> n2_t_vlmeval_dataset_image_base n7_kit_vlmeval_dataset___init__ --> n3_meval_dataset_utils___init__ n7_kit_vlmeval_dataset___init__ --> n0_evalkit_vlmeval_smp___init__ n2_t_vlmeval_dataset_image_base --> n0_evalkit_vlmeval_smp___init__ n2_t_vlmeval_dataset_image_base --> n6__mm_vlmevalkit_vlmeval_tools n3_meval_dataset_utils___init__ --> n8_val_dataset_utils_judge_util n3_meval_dataset_utils___init__ --> n4_ataset_utils_multiple_choice n8_val_dataset_utils_judge_util --> n0_evalkit_vlmeval_smp___init__ n4_ataset_utils_multiple_choice --> n1_alkit_vlmeval_utils___init__ n4_ataset_utils_multiple_choice --> n0_evalkit_vlmeval_smp___init__ n5_t_vlmeval_dataset_video_base --> n0_evalkit_vlmeval_smp___init__ classDef cyc stroke:#ef4444,stroke-width:2px; class n7_kit_vlmeval_dataset___init__ cyc;

Findings

MediumNo CI/CD pipeline detected - no automated build/test gate .github/ or CI config
LowSecret-shaped paths present; the code-health audit confirms or clears them eval_mm/vlmevalkit/.env, web_demos/minicpm-o_2.6/web_server/.env.development, web_demos/minicpm-o_2.6/web_server/.env.production

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

critical1

high4

medium4

low1

Rotate the credentials in the committed environment filecritical2 occurrences

eval_mm/vlmevalkit/.env

OPENAI_API_KEY, HUNYUAN_SECRET_KEY hold generated values

These values are loaded by the application at boot, so a generated value here is a working credential for something that runs.

Fix: Rotate each named credential, remove the file from the index, and replace it with a keys-only example.

Add a test suite; this repository has nonehigh158 occurrences

158 source files, no test files

Any change ships with no signal that existing behaviour still holds, so a regression reaches production undetected.

Fix: Add one test per public entry point, then a CI step that runs them.

Add a workflow that builds and tests this repositoryhigh

137 source files, no CI configuration

Every change merges with nobody having run the build once, which across this estate is 499 repositories.

Fix: Add a workflow running the project build and test command on push and pull_request.

Add a build gate for the deployable artifacts herehigh

web_demos/minicpm-o_2.6/web_server/Dockerfile

deployment artifacts with no workflow

Something in the repo is meant to reach a server and no automated step has ever validated it, which is the state of 125 repositories here.

Fix: Add a workflow that at least builds the image and validates the manifests.

Remove the committed .env and rotate what it holdshigh3 occurrences

eval_mm/vlmevalkit/.env

eval_mm/vlmevalkit/.env, web_demos/minicpm-o_2.6/web_server/.env.development, web_demos/minicpm-o_2.6/web_server/.env.production

A tracked .env is the most common route for a working key to reach a public clone, and it is the file the app actually loads, so the value is usually live.

Fix: git rm --cached the file, add it to .gitignore, rotate every credential it names, commit a .env.example with empty values.

Enable Dependabot or Renovatemedium

6 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium2 occurrences

web_demos/minicpm-o_2.6/web_server/Dockerfile

modelbest-registry-vpc.cn-beijing.cr.aliyuncs.com/modelbest/playground:20.10.0, modelbest-registry-vpc.cn-beijing.cr.aliyuncs.com/modelbest/playground:alpine

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Add a pre-commit secret gatemedium

a secret-shaped file is tracked and no repo-level gate is visible

Without a gate the same class of leak recurs on the next commit, so a leak finding is the symptom and this is the cause.

Fix: Add a pre-commit hook scanning staged content, and enable push protection.

Move large binaries to Git LFS or out of the repositorymedium30 occurrences

assets/minicpmv4_5/MiniCPM-V 4.5-8.26.mp4

30 blobs over 5MB: MiniCPM-V 4.5-8.26.mp4 45.0MB, v45_en_cot.gif 21.9MB, minicpmv2-cases_2.png 18.3MB

One repository here carries twenty blobs over 5MB including a 10.9MB spreadsheet, so every clone and every CI checkout pays for data nobody diffs.

Fix: Track those extensions with LFS, or move datasets to object storage and fetch them in a setup step.

Add the repository convention files this project lackslow3 occurrences

missing .editorconfig, .gitattributes, a formatter config

Without them one contributor's editor writes tabs into a Python file, a shell script commits with CRLF and fails in the container, and a notebook diff is unreviewable.

Fix: Add .editorconfig, .gitattributes with text=auto eol=lf, and a formatter config.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.