The Problem

Running language models on microcontrollers has been limited by SRAM constraints. The ESP32-S3 provides only 512KB of fast SRAM, yet this project runs a 28.9M parameter LLM that generates text at 9.88 tokens/second. The core tension: a model an order of magnitude larger than traditional microcontroller workloads must fit within 8MB flash and 8MB PSRAM without offloading to a server.

What This Does

This repository implements Per-Layer Embeddings (adapted from Google's Gemma 3n) to store 25M of 28.9M parameters in a flash-resident embedding table. The architecture partitions memory by access frequency:

  • SRAM (512KB): activations and norm weights, touched many times per token
  • PSRAM: the dense core and output head, read once per position
  • FLASH (16MB): the 25M-param embedding table, ~6 rows read per token (~450 bytes)

Key files enabling this: firmware/esp32_barista/esp32_barista.ino, firmware/esp32_barista/display.h, runtime/bpe_tokenizer.h, and runtime/llm.h. The scripts/generate_tokenizer_header.py and scripts/generate_vocab_headers.py tools convert the tokenizer vocab into C headers baked into firmware. The runtime/host_verify/ directory contains C verification utilities (tokenizer_conformance.c, bpe_tokenizer_verify.c) that validate the tokenizer implementation against the Python counterpart.

How It Is Wired

Execution starts at the Arduino sketch firmware/esp32_barista/esp32_barista.ino → initializes the BPE tokenizer from the generated header (runtime/bpe_tokenizer.h) → runs inference through the quantized model layers. The runtime/llm.h defines the inference entry point; runtime/host_verify/ppl.c and runtime/host_verify/verify.c provide reference implementations for perplexity calculation and token verification.

The model weights flow through this path: scripts/fetch_model.sh downloads and verifies the model (SHA-256 + byte size checks against metadata.json) → installs into artifacts/<model>/ → scripts/deploy.sh runs the generator tools to produce C headers → uv fetches a pinned wheel → firmware compiles and flashes. The runtime/quantize.py (Python) and src/quantize.py handle 4-bit quantization, producing the 14.9MB model artifacts.

What leaves this process: serial text output via UART to the display wired to the chip. Nothing is sent to a network—connectivity is nonexistent by design. The widest blast radius is the tokenizer header generation (scripts/generate_tokenizer_header.py); if that produces an incorrect header, the entire firmware build fails or produces garbage output.

How To Use It

Setup: Install ESP32-S3 Arduino core and Python dependencies:

# From repo root
uv sync  # installs deps from pyproject.toml + uv.lock

Configuration: Model selection is baked into the firmware compile; deploy scripts/fetch_model.sh barista downloads the Barista model (espresso QA) or scripts/fetch_model.sh tinystories downloads the story generation model. Both verify SHA-256 and byte size before installing to artifacts/<model>/.

Running it: Two-step process from repo root:

scripts/fetch_model.sh barista   # download + verify + install to artifacts/
scripts/deploy.sh barista        # generate headers, run gates, compile, flash

The same commands work for tinystories. The board holds one model at a time; deploying a new model replaces the previous one.

Real-World Use

This fits as a local, offline text generator for embedded interfaces—e.g., a coffee machine interface running espresso QA (Barista model), a children's story device, or a voice assistant with no cloud dependency. The code snippet below shows the minimal Arduino loop for text generation:

// firmware/esp32_barista/esp32_barista.ino loop()
void loop() {
  if (Serial.available()) {
    String prompt = Serial.readStringUntil('\n');
    float latency = infer(prompt.c_str());  // defined in runtime/llm.h
    Serial.print("Tokens/s: "); Serial.println(1.0 / latency);
  }
}

Code Health & Issues

  • [Medium/SDLC] No CI/CD pipeline detected—no automated build/test gate. The .github/ directory is absent; every compile/flash cycle is manual.
  • [Low/Risk] Dependencies declared without a lockfile in pyproject.toml (the uv.lock exists but is not referenced as the lockfile by the build config). Non-reproducible builds if uv is not pinned to the same version across environments.

The Bottom Line

This is a technically impressive architecture hack that makes a 28.9M parameter LLM run on an ESP32-S3 by keeping 25M parameters in flash and sampling rows incrementally. The Per-Layer Embeddings approach is the real innovation here—not the model quality (TinyStories/Barista are simple story generators), but the memory layout that lets a microcontroller run what should be a embedded-server-scale model. Use this if you need offline, on-device text generation on constrained hardware and accept the model's narrow capability scope. The lack of CI and lockfile coupling is a practical SDLC gap for any team beyond a solo experimenter.