The Problem

Researchers who want to experiment with LLM training often spend hours tweaking data pipelines, model code, and hyper‑parameters manually. The effort required to set up reproducible runs on a single GPU and iterate quickly becomes a bottleneck, especially when the goal is rapid, autonomous exploration.

What This Does

autoresearch supplies a minimal, single‑GPU training stack derived from Karpathy’s nanochat implementation. The repository consists of only ten files; the core logic lives in train.py, while prepare.py handles one‑time data download, BPE tokenizer training, and utility helpers such as list_parquet_files and make_dataloader. A human edits program.md to give high‑level instructions to an AI agent; the agent repeatedly mutates train.py, runs a 5‑minute training job, and evaluates validation bits‑per‑byte (val_bpb). The workflow is deliberately bounded: fixed wall‑clock budget, single GPU, and a single entry point (train.py) that the agent may modify without touching the surrounding infrastructure.

How It Is Wired

Execution starts with the CLI command uv run train.py. train.py defines the GPT model (classes for CausalSelfAttention, Block, etc.) and the training loop. The most connected functions are:

  • forward → norm (called 6 times) – core model forward pass.
  • __init__ → has_ve (called 2 times) – model initialization.

train.py imports prepare.py only for data utilities; the import graph shows a single edge from train to prepare.

prepare.py owns all filesystem interaction:

  • download_data / download_single_shard – network fetches (the only outbound network call).
  • list_parquet_files, text_iterator, train_tokenizer – read/write of local parquet files and tokenizers.

The internal call graph contains 42 intra‑repo edges. Functions with the widest blast radius are:

  • list_parquet_files – called from three places (text_iterator, train_tokenizer, _document_batches).
  • norm – invoked six times per forward pass, affecting every training step.

No circular dependencies exist, so changes in prepare.py do not cascade back into train.py. However, deep nesting (max indentation depth 8) in both train.py and prepare.py makes the control flow harder to follow, increasing cognitive load for any modification.

How To Use It

# 1. Clone the repo
git clone https://github.com/moses-y/autoresearch
cd autoresearch

# 2. Install uv (if not already present)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 3. Install dependencies as defined in pyproject.toml
uv sync

# 4. One‑time data download and tokenizer training (≈2 min)
uv run prepare.py

# 5. Run a single 5‑minute training experiment
uv run train.py

The repository does not define additional configuration files or environment variables; all required settings are hard‑coded in prepare.py (e.g., data URLs) and train.py (model hyper‑parameters).

Real‑World Use

A research team could wrap the above steps in a scheduler that repeatedly launches uv run train.py under different program.md prompts. After each run, the agent reads the generated val_bpb from the console output, decides whether to keep the mutation, and writes a new version of train.py. The human operator only needs to adjust program.md to steer the agent’s exploration strategy.

Code Health & Issues

  • High – Unsafe deserializer – prepare.py uses pickle.load() (and yaml.load()), which can execute arbitrary code if the data source is compromised. Replace with pickle.safe_load‑compatible formats (e.g., JSON or a pure‑data serialization).
  • Medium – Missing automated dependency updates – Only pyproject.toml is present; no Dependabot or Renovate configuration. Add .github/dependabot.yml to keep libraries patched.
  • Medium – No test suite – No tests/ directory; code paths are unverified by automated tests.
  • Medium – No CI pipeline – No .github/workflows/ or other CI config, so builds are not automatically validated.
  • Medium – No license file – Usage and redistribution rights are unclear.
  • Low – No lockfile – pyproject.toml alone makes reproducible installs harder; consider committing uv.lock.
  • High – Deep nesting – Both train.py and prepare.py reach 8 levels of indentation, reducing readability and increasing risk of bugs. Refactor with early returns or helper functions.

The Bottom Line

autoresearch delivers a compact, reproducible single‑GPU training loop that can be autonomously mutated by an AI agent, making it useful for rapid prototyping of LLM experiments. The codebase is small and well‑isolated, but it lacks test coverage, CI, a license, and safe serialization practices, and its deep nesting hampers maintainability. Teams comfortable adding their own testing and CI scaffolding can adopt it for controlled autonomous research; otherwise, the repository needs hardening before production use.