OpenCoder llm

The Problem Teams that train or evaluate code‑focused LLMs need a reproducible benchmark harness and a lightweight finetuning pipeline. Existing scripts are scattered, often duplicated, and lack a single, well‑documented entry point, making it hard to integrate them into CI or automated research workflows.

What This Does The repository ships two related pieces:

1. OpenCodeEval – a self‑contained evaluation framework (45 files) that implements the BigCodeBench, HumanEval, LeetCode, MBPP and MBPPPlus benchmarks. Core logic lives in OpenCodeEval/src/benchmark/*.py and OpenCodeEval/src/eval/*.py.
2. sft – a minimal finetuning helper (sft/tools/finetune.py) that builds prompts, tokenises data and calls a model’s generate method.

Key files:

• OpenCodeEval/src/main.py – CLI driver that parses args, spawns workers and writes results (write_jsonl).
• OpenCodeEval/src/backend/base.py – defines Generator and the generic generate wrapper used by both the OpenAI and vLLM back‑ends.
• OpenCodeEval/src/utils.py – helpers for prompt construction (make_chat_prompt, refine_text) and streaming results (stream_jsonl).

How It Is Wired

Execution starts at OpenCodeEval/src/main.py:40 → main. main builds a multiprocessing pool, then calls generate (via the selected backend) for each benchmark task. The call chain of interest is:

main → generate → is_chat / make_chat_prompt
       → refine_text (6 distinct callers)
       → stream_jsonl (5 callers)
       → write_jsonl (5 callers)

The backend (OpenCodeEval/src/backend/openai.py or vllm.py) implements Generator.generate, which ultimately invokes self.model.generate – the only place the code reaches out of the repository (model inference). File‑level responsibilities that dominate the call graph:

No circular imports are present; each module is isolated, simplifying refactoring. The most “blast‑radius” functions are refine_text and get_task, each called from 4–6 locations, so changes there affect many benchmark paths.

How To Use It

# 1. Clone the repo
git clone https://github.com/moses-y/OpenCoder-llm.git
cd OpenCoder-llm

# 2. Install Python dependencies (no lockfile, pip install from the two requirement files)
pip install -r OpenCodeEval/requirements-eval.txt
pip install -r sft/requirements.txt

# 3. (Optional) Prepare data – the repository already contains JSONL benchmark files
#    Large files (HumanEvalPlus.jsonl, BigCodeBench.jsonl) are >5 MiB; consider Git LFS if needed.

# 4. Run the evaluation driver
python OpenCodeEval/src/main.py --backend openai   # or --backend vllm

The --backend flag selects OpenCodeEval/src/backend/openai.py or vllm.py. Results are written to a JSONL file by write_jsonl (called from main). No additional configuration files or environment variables are required by the code itself.

Real‑World Use A research team can plug this framework into an automated experiment pipeline: after training a new code LLM checkpoint, they invoke the same python …/main.py command inside a CI job, capture the generated JSONL, and feed it to downstream analysis scripts. Because the entry point is a single script and the benchmark tasks are pure functions, swapping the model or adding a new benchmark only requires implementing a new Backend subclass.

Code Health & Issues

• High – Missing CI – 21 source files, no .github/workflows or other CI config. Fix: add a GitHub Actions workflow that runs pip install -r … && python -m pytest on push/PR.
• Medium – No Dependabot – Two manifest files (requirements-eval.txt, requirements.txt) but no automated update bot. Fix: add .github/dependabot.yml for the pip ecosystem.
• Medium – Large binaries not in LFS – OpenCodeEval/src/data/HumanEvalPlus.jsonl (10.8 MiB) and BigCodeBench.jsonl (6.6 MiB). Fix: track these with Git LFS or host them externally and download in a setup step.

Additional observations: the repository contains a .gitignore, a LICENSE, and three test files, but the test suite is not hooked into any CI. No Dockerfile or reproducible environment definition is present.

The Bottom Line OpenCoder‑llm provides a clean, modular evaluation harness and a tiny finetuning helper that can be dropped into existing code‑LLM research pipelines. The code is well‑segmented, but duplicated blocks and deep nesting raise maintenance overhead, and the lack of CI, lockfiles, and LFS for data hampers reproducibility. It is suitable for teams comfortable managing their own CI and willing to refactor the repeated helper logic.