The Problem

Reproducing a machine-learning paper's implementation from its PDF or LaTeX source is slow, error-prone manual work. PaperCoder automates that conversion: it takes a scientific paper and produces a working code repository, using a multi-agent LLM pipeline.

What This Does

PaperCoder runs a three-stage pipeline: planning, analysis, and code generation, each handled by a separate agent (codes/1_planning.py, codes/2_analyzing.py, codes/3_coding.py). The pipeline accepts either a PDF converted to JSON or raw LaTeX source, and emits a repository under outputs/<paper>_repo.

Two backends exist: OpenAI API (*_llm.py variants excluded) and local vLLM serving, selected by which shell script you run from scripts/. The repo also ships the Paper2Code benchmark dataset under data/paper2code/ and an evaluation script (codes/eval.py).

How It Is Wired

Execution starts at main in codes/0_pdf_process.py:19, which reaches 14 functions. The pipeline is orchestrated by shell scripts (scripts/run.sh, run_latex.sh, run_llm.sh), not by Python imports — the import graph shows 0 edges between the 13 internal modules. That means the modules are coupled through the filesystem and the scripts, so changing the order of stages means editing a shell script, and there is no static type or import safety net between stages.

The single hub is codes/utils.py: 18 functions, called from 1 other file, and the most-called functions in the repo (content_to_json, extract_planning, print_log_cost) live there. content_to_json chains through content_to_json2/3/4 — a deep call chain that is part of the 374 duplicated 6-line blocks the analysis found across 11 files. Three functions read or write files; three call a model for inference. The shortest path from an entry point to leaving the process is main -> api_call, which hits client.chat.completions.create — two hops from start to the network call.

The _llm file pairs (1_planning_llm.py, 2_analyzing_llm.py, 3_coding_llm.py) mirror the API versions but route through run_llm for vLLM. codes/4_debugging.py applies fixes back to files via parse_and_apply_changes.

How To Use It

Per the README:

pip install openai
export OPENAI_API_KEY="<OPENAI_API_KEY>"
cd scripts
bash run.sh

For local models: pip install vllm, then cd scripts && bash run_llm.sh (defaults to deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct). The example paper is "Attention Is All You Need". Input papers are configured via environment variables in the shell scripts; the README documents switching to your own paper by modifying those. Estimated cost for the o3-mini run: $0.50–$0.70.

Real-World Use

A research team wants to reproduce a paper's results without hand-writing the implementation. They convert the PDF to JSON using the documented s2orc-doc2json flow, set OPENAI_API_KEY, run scripts/run.sh, and get a generated repository under outputs/<paper>_repo plus planning/analysis/coding artifacts they can audit before trusting the code.

Code Health & Issues

Static analysis (not opinion) found the following:

  • High – Duplicated code blocks – 374 repeated 6-line blocks across 11 files, including codes/1_planning.py, codes/2_analyzing.py, codes/3.1_coding_sh.py, codes/3_coding.py. Fix: extract shared helpers.
  • Medium – Deep nesting – max indentation depth 6 in codes/4_debugging.py and codes/utils.py. Fix: guard clauses and early returns.
  • High – No test suite – 13 source files, zero tests; every change ships with no regression signal.
  • High – No CI – no build or test gate on any change.
  • Medium – No dependency update bot – requirements.txt exists but no Dependabot/Renovate config.
  • Medium – Large binary committed – data/paper2code/paper2code_data.zip (5.0MB) bloats every clone; move to LFS or object storage.
  • Low – No lockfile – requirements.txt without pinned transitive deps means non-reproducible builds.

The Bottom Line

The pipeline design is sound and the README is genuinely usable. The weaknesses are engineering hygiene: no tests, no CI, heavy code duplication, and a 5MB dataset committed to git. Use it for prototyping paper-to-code conversion, not as a foundation to build on without first adding a test suite and CI.