The Problem

Tokenizing massive text corpora is a bottleneck for large‑scale language‑model pipelines. Existing Python tokenizers (HF tokenizers, tiktoken) are fast but still limited to a few hundred MB/s, forcing costly preprocessing steps.

What This Does

gigatoken provides a Rust‑backed tokenizer library exposed to Python. The core lives in src/ (src/lib.rs, src/main.rs) and is compiled via maturin. Compatibility shims in gigatoken/_hf_compat.py, gigatoken/_tiktoken_compat.py, and gigatoken/_load/ let a Tokenizer object behave like HuggingFace or tiktoken tokenizers while delivering GB/s throughput.

Key files:

  • gigatoken/_tokenizer.py – public API (Tokenizer, encode_batch, encode_files).
  • gigatoken/_load/hf.py, gigatoken/_load/tiktoken.py – model‑specific loader helpers.
  • benchmarks/compare/measure.py – a runnable benchmark that drives the whole stack.

The repository is a portfolio of 8 self‑contained projects (benchmarks, notebooks, examples, profiling, etc.) that all reuse the same tokenizer core.

How It Is Wired

Execution starts in benchmarks/compare/measure.py:main (line 144). The main function parses CLI args, then calls cpu_label (a subprocess call) and proceeds to:

  1. Load a tokenizer via gigatoken._load.hf.load_hf or load_tiktoken.
  2. Build a Tokenizer object (gigatoken._tokenizer.Tokenizer).
  3. Call encode_files which internally invokes Tokenizer.encode_batch.

The internal call graph shows the hottest symbols:

  • Tokenizer – referenced by 42 distinct callers.
  • encode_batch – also 42 callers.
  • _ids – 28 callers.

These functions sit in gigatoken/_tokenizer.py, making it the blast‑radius hub; any change ripples through most of the codebase.

Circular imports involve gigatoken/_hf_compat.py, gigatoken/_tokenizer.py, and gigatoken/_tiktoken_compat.py. The cycle forces runtime import‑time side effects and hampers refactoring.

Large, deeply nested files such as src/bpe/tiktoken.rs (1 773 lines, nesting depth 9) and src/pretokenize/fast/o200k_family.rs dominate the Rust side; they are called by many high‑level functions (Tokenize, batch, pretokenize).

External interactions are limited to:

  • Subprocess call cpu_label → sysctl (CPU model query).
  • File I/O in benchmarks, examples, and the loader modules (reading model files, token vocabularies).
  • No network calls from the core library; only test fixtures download data via tests/conftest.py.

How To Use It

# Clone the repository
git clone https://github.com/moses-y/gigatoken
cd gigatoken

# Build and install the Python package (uses maturin under the hood)
pip install .
# or, with uv (if preferred)
uv pip install .

The package installs a console script gigatoken that forwards to gigatoken._cli._main. For a quick benchmark:

# Example from the README
uvx --with tokenizers gigatoken bench openai-community/gpt2 owt_train.txt \
    --validate --doc-separator "<|endoftext|>"

Running the supplied benchmark directly:

python benchmarks/compare/measure.py \
    --model openai-community/gpt2 \
    --input owt_train.txt \
    --validate

The examples/ folder contains ready‑to‑run scripts (quickstart.py, encode_files.py) that illustrate the public API.

Real‑World Use

A data‑engineering pipeline can replace its HF tokenizer call:

import gigatoken as gt

tokenizer = gt.Tokenizer("meta-llama/Meta-Llama-3-8B")
tokens = tokenizer.encode_files(gt.TextFileSource(["large_corpus.txt"]))
# tokens now stream at >8 GB/s on a modern 16‑core CPU

The token IDs are byte‑compatible with HF tokenizers, so downstream model code needs no changes.

Code Health & Issues

Measured findings (static analysis)

  • HIGH – Import cycle – gigatoken/_hf_compat.py, gigatoken/_tokenizer.py, gigatoken/_tiktoken_compat.py (mutual imports).
  • HIGH – Deep nesting – up to 9 levels in src/bpe/tiktoken.rs, src/pretokenize/fast/olmo3.rs.
  • HIGH – Duplicated code blocks – identical 6‑line snippets across benches and notebooks.
  • HIGH – Oversized files – src/bpe/tiktoken.rs, src/pretokenize/fast/o200k_family.rs, src/batch.rs (>1 700 lines each).
  • MEDIUM – Broad exception handling – generic except: in _load/hf.py, _hf_compat.py, notebooks/data_view.py.
  • MEDIUM – Hub module – gigatoken/__init__.py is imported by 20 modules; volatility here has high blast radius.

Code‑health audit (CI/ops)

  • HIGH – Pin GitHub Actions – .github/workflows/*.yml uses PyO3/maturin-action@v1.
  • HIGH – CI never runs tests – workflows list no test step despite 31 test files.
  • MEDIUM – No Dependabot/Renovate – no auto‑update config for Cargo or pip dependencies.
  • MEDIUM – No dependency‑vulnerability scan – CI lacks dependency-review-action or similar.
  • MEDIUM – Checkout persists token – persist-credentials missing in wheels.yml.
  • LOW – No job timeout – CI jobs lack timeout-minutes.

No license, lockfile, or secret issues were detected beyond the above.

The Bottom Line

gigatoken delivers the advertised GB/s tokenization speed and drop‑in compatibility, but the codebase contains several maintainability pain points: import cycles, very large/complex Rust files, and incomplete CI. It is suitable for teams that need raw throughput and are prepared to manage the technical debt or contribute fixes.