The Problem

Large language models (LLMs) need a realistic, reproducible way to prove they can read a real GitHub issue, understand the surrounding code, and emit a correct patch. Researchers and product teams currently hand‑craft small toy examples, which do not reflect the complexity of actual repositories, dependency management, or build environments.

What This Does

SWE‑bench supplies a curated collection of real‑world GitHub issues together with the exact code snapshots needed to evaluate an LLM’s patch‑generation ability. The core data lives under swebench/resources/ and is accessed via the datasets library (see README snippet). The evaluation harness lives in swebench/harness/, where Docker images are built (docker_build.py, docker_utils.py) and test scripts are generated (test_spec/*.py). Model‑side utilities for dataset creation and retrieval sit in swebench/inference/, e.g. make_datasets/utils.py and llamao/modeling_flash_llama.py.

How It Is Wired

Execution starts at the CLI entry point swebench/collect/build_dataset.py:main. From there:

  1. Dataset Construction – main calls create_instance, is_valid_pull, is_valid_instance, and has_test_patch. These functions read the repository snapshots and filter out unsuitable issues (filesystem I/O).
  2. Docker Harness – build_image (in harness/docker_build.py) is invoked to assemble a container that mirrors the original environment; it uses docker_utils.exec_run_with_timeout for command execution and remove_image for cleanup.
  3. Test Spec Generation – test_spec.py creates per‑issue scripts (setup_env_script, eval_script, install_repo_script) that the container will run. The module harness/constants/* supplies language‑specific environment definitions; it is the most‑connected hub (23 importers) and therefore a high‑blast‑radius area.
  4. Evaluation – run_evaluation.py (or its Modal variant) reads the generated spec, launches the Docker container, streams logs via log_parsers/*, and collects the model’s output. The call chain run_instance_modal → exec → EvaluationError/TestOutput occurs five times, showing the centrality of the modal runner.
  5. External Effects – Throughout the pipeline, the code performs: Filesystem – reading/writing issue data, Dockerfiles, and temporary logs (e.g., merge_results → os.remove). External Commands – Docker CLI invocations and model‑loading scripts (21 functions). Network – utils.call_api fetches issue metadata from GitHub (8 functions). Inference – the LLM is called via inference/run_llama.py (4 functions).

No circular import cycles were detected, but the hub module swebench/harness/constants/__init__.py imports many language‑specific constant files, so changes there can ripple widely.

How To Use It

# Clone the repository (use the exact URL from the prompt)
git clone https://github.com/moses-y/SWE-bench
cd SWE-bench

# Install in editable mode (pyproject.toml defines the package)
pip install -e .

# Build the Docker evaluation harness (requires Docker Engine)
python -m swebench.collect.build_dataset   # runs the main entry point

No additional configuration files are required; Docker must be installed per the guide in docs/guides/docker_setup.md. For cloud‑based evaluation, the Modal entry point swebench/harness/modal_eval/run_evaluation_modal_entrypoint.py can be invoked instead of the local Docker flow.

Real‑World Use

A product team can integrate SWE‑bench into a CI pipeline to automatically generate a test set for a new LLM release:

# .github/workflows/eval.yml
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install
        run: pip install -e .
      - name: Build dataset & run evaluation
        run: python -m swebench.collect.build_dataset

The workflow builds the Docker images, runs the model against each issue, and stores the resulting patches for downstream analysis.

Code Health & Issues

Measured findings (static analysis):

  • HIGH – Pin GitHub Actions to commit SHAs (.github/workflows/*.yaml).
  • HIGH – Add a lockfile beside pyproject.toml.
  • MEDIUM – Declare least‑privilege permissions for GITHUB_TOKEN (.github/workflows/pytest.yaml).
  • MEDIUM – Enable Dependabot or Renovate (.github/dependabot.yml missing).
  • MEDIUM – Add a dependency‑vulnerability scan step to CI.
  • MEDIUM – Set persist-credentials: false on checkout (deploy-docs.yml).
  • LOW – Add timeout-minutes to workflow jobs (deploy-docs.yml).

No critical or blocker issues were found. Tests and CI are present, a LICENSE file exists, and no secrets are committed.

The Bottom Line

SWE‑bench provides a well‑structured, Docker‑based benchmark for measuring LLM patch quality on authentic GitHub issues. The codebase is functional but includes several maintenance‑level concerns (broad exception handling, deep nesting, duplicated blocks) and a handful of CI hygiene gaps that should be addressed before production use. It is suitable for research teams and organizations that need a realistic evaluation sandbox and are comfortable managing Docker‑based workloads.