The Problem
Training models that can dynamically allocate compute across tokens and reasoning depth is an open research problem. Standard transformers use a fixed number of layers per forward pass, which wastes compute on easy tokens and under-serves hard ones. Recurrent-depth transformers address this by looping a block of layers a variable number of times, but no open-source implementation existed to experiment with the architecture. OpenMythos fills that gap with a research-grade, configurable reference implementation.
What This Does
OpenMythos implements a Recurrent-Depth Transformer (RDT) with three stages: Prelude transformer blocks, a looped Recurrent Block, and a Coda. The core logic lives in openmythos/main.py, with modular components in openmythos/moda.py (attention, MoE, and recurrent injection) and openmythos/tokenizer.py. Attention is switchable between Multi-head Latent Attention (MLA) and Grouped Query Attention (GQA), configurable via MythosConfig.
The feed-forward network uses a sparse Mixture-of-Experts with routed and shared experts, controlled through config fields like nexperts, nsharedexperts, and nexpertspertok. Pre-configured model scales from 1B to 1T parameters are defined in openmythos/variants.py. The recurrent.injection.getA() method exposes the spectral radius of the recurrent weight matrix, which must stay below 1 for stability.
How To Use It
Setup: Install via pip, as documented in the README:
pip install open-mythos
Configuration: All configuration is done through MythosConfig in openmythos/main.py. The README shows a full config with attention-type-specific parameters. No environment variables or external config files are required.
Running it: Instantiate the model and pass token IDs directly. The README example in example.py shows the full workflow: create a config, build OpenMythos(cfg), run a forward pass with model(ids, nloops=4), and generate with model.generate(ids, maxnewtokens=8, nloops=8).
Real-World Use
A research team exploring compute-adaptive inference could use this to test whether looping a recurrent block improves accuracy on hard tokens without retraining. The config-driven design makes ablation studies straightforward: compare MLA vs. GQA, vary maxloopiters from 1 to 8, and measure the parameter/compute trade-off. The training/3bfinewebedu.py script provides a starting point for fine-tuning a 3B variant on educational web data.
Code Health & Issues
Med - No CI/CD pipeline - no .github/ or CI config exists, so there is no automated gate for tests or linting. The 4 test files in tests/ will only run if someone invokes them manually. Low - No dependency lockfile - pyproject.toml and requirements.txt declare dependencies but no lockfile pins exact versions, so builds are not reproducible. Low - Sparse test coverage - testmain.py, tests/testropedebug.py, and tests/testtokenizer.py exist, but the README's spectral radius check is not covered in the test suite. The testrope_debug.py filename suggests a debug-oriented test rather than a systematic suite. Low - Single maintainer risk - this is a fork of kyegomez/OpenMythos (14.8k stars), but this repo has 0 stars and no contributor activity visible. It is a snapshot, not a maintained project.
The Bottom Line
OpenMythos is a solid, well-structured reference implementation of an interesting architecture, with clean modularity and sensible configuration. It is suitable for researchers who want to experiment with recurrent-depth transformers and have the patience to run tests manually. Production teams should wait for CI and lockfile hygiene before building on it.