The Problem

Running large language models (70B+ parameters) typically requires multiple high-end GPUs with 80GB+ of VRAM each. AirLLM addresses this by enabling inference of 70B models on a single 4GB GPU card without quantization, distillation, or pruning, and even claims support for Llama3.1 405B on 8GB VRAM. This makes large-model inference accessible to developers without expensive hardware.

What This Does

The core package in air_llm/ implements memory-efficient inference by loading model layers to GPU one at a time and keeping the rest on CPU/disk. The air_llm/airllm/airllm_base.py file defines the AirLLMBaseModel class with methods like load_layer_to_cpu, move_layer_to_device, and set_layers_from_layer_names that orchestrate this layer-by-layer execution. Model-specific subclasses (airllm_llama_mlx.py, airllm_baichuan.py, airllm_qwen.py, etc.) handle architecture variations.

The repo also contains several independent projects: anima_100k/ (long-context training), rlhf/ (DPO fine-tuning), training/ (QLoRA), and eval/ (evaluation notebooks). These are separate efforts sharing the AirLLM ecosystem rather than one integrated codebase.

How It Is Wired

Execution starts at air_llm/airllm/__init__.py (imports 12 modules). The entry point for users is typically AutoModel in auto_model.py, which routes to the correct model class. The central hub is airllm_base.py — 9 modules import it, making it the highest-fan-in module. It defines init_model and set_layers_from_layer_names, which most model implementations call.

The persist/ directory contains model_persister.py and its subclasses (mlx_model_persister.py, safetensor_model_persister.py), which handle model serialization. These three files participate in a circular import cycle, complicating refactoring.

Key control flow: AirLLMBaseModel.__init__ → init_model → loads config → set_layer_names_dict → generation loop calls load_layer_to_cpu/move_layer_to_device per layer, with utils.py handling clean_memory and load_layer between steps. The utils.py module has high branching density (98 branch points over 286 lines).

How To Use It

git clone https://github.com/moses-y/airllm
cd airllm
pip install -r requirements.txt
pip install -e air_llm/

The README documents usage via the airllm pip package:

from airllm import AutoModel
model = AutoModel.from_pretrained("meta-llama/Llama-2-70b-hf")
output = model.generate(input_ids)

Configuration is minimal — model paths and generation parameters are passed directly. No environment variables or config files are required. Example notebooks exist in air_llm/examples/.

Real-World Use

A developer with a single consumer GPU (4–8GB VRAM) wants to run Llama-3-70B for prototyping. AirLLM loads the model from HuggingFace, keeps embedding layers in GPU memory, and streams transformer layers through GPU memory during generation. The trade-off is speed: each layer must be loaded from disk/CPU for every forward pass, making inference significantly slower than with full GPU residency.

Code Health & Issues

Static analysis (20 findings: 8 high, 12 medium) identified:

  • High - Deep nesting - airllm_base.py, utils.py, airllm_llama_mlx.py have max indentation depth of 22, making control flow hard to follow.
  • High - Import cycle - persist/model_persister.py, mlx_model_persister.py, safetensor_model_persister.py are mutually reachable.
  • High - Duplicated code - 683 repeated 6-line blocks across 10 files.
  • Medium - Broad exception handling - rlhf/qlora_dpo.py, training/qlora.py use bare except.
  • Medium - Oversized files - anima_100k/modeling_flash_llama.py (761 lines), longer_training.py, rlhf/qlora_dpo.py.

No CI pipeline, no lockfile, and no Dockerfile. Tests exist (10 files) but aren't automated.

The Bottom Line

AirLLM is a practical solution for running large models on constrained hardware, with a proven track record (3k+ stars upstream). The code works but has maintainability issues — deep nesting, import cycles, and duplication. Use it for inference workloads where hardware constraints matter more than speed; expect to invest effort if you need to modify core logic.