The Problem

Current multimodal LLM benchmarks mix perception with reasoning and knowledge, making it hard to isolate visual‑perception failures. Teams need a focused, reproducible suite that measures “atomic” perception capabilities across models.

What This Does

The repo implements the PerceptionBench evaluation harness. The only executable code lives in eval/eval.py, which loads a question set, builds prompt messages, calls an LLM via the OpenAI API, and judges the short answer against a reference. Supporting assets include the prompt template (eval/judge_prompt.txt), visual assets for the README, and the benchmark PDF (paper/PerceptionBench.pdf). Dependencies are listed in requirements.txt (e.g., openai>=1.0.0).

How It Is Wired

Entry point – python eval/eval.py (the file ends with a main() call).

  1. main() parses CLI arguments (not shown) and iterates over the benchmark items, invoking run_one() for each.
  2. run_one() → build_messages() constructs the chat history, then calls chat_with_retry() to send the request to the model.
  3. chat_with_retry() wraps the OpenAI call in a retry loop, handling rate‑limit errors. The actual inference is performed by the OpenAI client (outside the repo).
  4. The response is passed to judge_general_qa(), which normalizes escape sequences via normalize_escape(), decodes the judge output with decode_judge(), and determines pass/fail via is_failed().
  5. Results are written to a CSV/JSON file (file I/O occurs inside run_one() and judge_general_qa()).

All eight internal call edges stay within eval/eval.py; no other modules are imported. The file therefore owns the full evaluation flow, file I/O, and the single external dependency on the OpenAI inference endpoint.

How To Use It

# Clone the original repository
git clone https://github.com/moses-y/PerceptionBench
cd PerceptionBench

# Install Python dependencies
python -m pip install -r requirements.txt

# Copy example environment and set your OpenAI key
cp .env.example .env
# edit .env to add OPENAI_API_KEY=<your-key>

# Run the benchmark (default settings)
python eval/eval.py

If the script expects additional arguments (e.g., model name or output path), they are documented in the README.md or displayed by python eval/eval.py --help. No Dockerfile, Makefile, or custom build steps are present.

Real‑World Use

A research team can integrate the harness into an automated model‑evaluation pipeline:

import subprocess, os

os.environ["OPENAI_API_KEY"] = "sk-..."
subprocess.run(["python", "eval/eval.py", "--model", "gpt-4o-mini", "--out", "results.csv"])

The generated CSV can be ingested by downstream analytics dashboards to track perception capability trends across model releases.

Code Health & Issues

  • Medium – Deep nesting – eval/eval.py has a maximum indentation depth of 6, making control flow hard to follow. Refactor with early returns or guard clauses.
  • Medium – Dependabot missing – Only requirements.txt is present; no Dependabot or Renovate config. Add .github/dependabot.yml to automate security updates.
  • Medium – Large binary – paper/PerceptionBench.pdf (10.2 MB) exceeds the 5 MB threshold. Move to Git LFS or external storage.
  • Medium – No tests – Repository contains no test files; code paths are unverified.
  • Medium – No CI pipeline – No .github/workflows or other CI configuration.
  • Low – No lockfile – Dependencies are not pinned; reproducibility depends on the latest package versions.

The Bottom Line

PerceptionBench provides a lightweight, single‑file evaluation harness that isolates visual‑perception performance for multimodal LLMs. The code is functional but minimally structured, with a deep nesting issue and missing production safeguards (tests, CI, dependency automation). It is suitable for research teams that need a quick, reproducible benchmark and are comfortable adding their own testing and CI layers.