The Problem

LLM tokens carry cost, and standard JSON is verbose for prompt input. Uniform arrays of objects—common in LLM contexts—inflate token counts because each object repeats keys like "id", "name", "companion". The repository acknowledges this: the README benchmarks JSON against TOON and YAML, showing TOON's compactness goal for LLM-facing data.

What This Does

TOON is a compact, human-readable encoding of JSON that minimizes tokens while preserving structure. It combines YAML-style indentation for nesting with a CSV-style tabular layout for uniform arrays, making it easier for models to parse and validate. The core library lives in packages/toon/, with packages/toon/src/index.ts as the main entry point, packages/toon/src/encode/normalize.ts handling value normalization, and packages/toon/src/decode/decoders.ts implementing the parser/evaluator. A CLI wrapper at packages/cli/src/index.ts provides run, encode, and decodeStream commands. Benchmarks reside in benchmarks/src/, notably benchmarks/src/questions/index.ts which drives evaluation task generation.

How It Is Wired

Execution starts at run in packages/cli/src/index.ts:130, which reaches 122 functions and is the only entry point called from outside the repo. From there, control flows through encode (packages/toon/src/index.ts:48, reaches 45 functions, called from 4 places) and decodeStream (packages/toon/src/index.ts:208, reaches 51 functions, called from 2 places). The internal call graph has 516 resolved edges; generateStructureQuestions in benchmarks/src/questions/index.ts:27 is a major hub, calling id, getId, prompt, groundTruth, type, dataset, answerType, and build exactly 25 times each. benchmarks/src/questions/utils.ts defines createIdGenerator, id, prompt, groundTruth, and type, and is called from 11 other files. The hub modules—benchmarks/src/types.ts, packages/toon/src/types.ts, and packages/toon/src/constants.ts—have 17 dependent modules, so changes ripple widely. A run trace ends at encodeToToon which calls fsp.writeFile, the only filesystem write path traced outside the process.

How To Use It

Setup: Install dependencies with pnpm install (the repo uses pnpm, as shown in package.json and pnpm-workspace.yaml). The workspace comprises packages/toon and packages/cli.

Configuration: No environment variables or keys are required; TOON is a pure format transformation.

Running it: To encode JSON to TOON, invoke the CLI:

pnpm cli encode --input input.json --output output.toon

or use the API:

import { encode } from '@toon-format/toon';
const toon = encode(yourJsonData);

The CLI entry point is packages/cli/src/index.ts:130 (the run function); the core encode function lives in packages/toon/src/index.ts:48.

Real-World Use

A prompt engineer preparing few-shot examples for an LLM can replace verbose JSON with TOON to reduce token count and improve model adherence to structure. For instance, a uniform array of hike records:

[
  {"id":1,"name":"Blue Lake Trail","distanceKm":7.5,"elevationGain":320,"companion":"ana","wasSunny":true},
  {"id":2,"name":"Ridge Overlook","distanceKm":9.2,"elevationGain":540,"companion":"luis","wasSunny":false}
]

encodes to TOON as:

1  Blue Lake Trail    7.5  320  ana  true
2  Ridge Overlook     9.2  540  luis false

The encode function in packages/toon/src/index.ts handles this transformation losslessly, and the decoder reconstructs the original JSON. This is useful when every token counts toward cost, and the data conforms to TOON's sweet spot: uniform arrays of objects with the same shape.

Code Health & Issues

The static analysis identified 16 findings across 4 kinds:

  • [HIGH/clarity] Duplicated code blocks – 33 repeated 6-line blocks across benchmarks/src/constants.ts, benchmarks/src/datasets.ts, packages/cli/test/json-from-events.test.ts, packages/toon/test/decodeStream.test.ts, and others. Fix: extract shared helpers.
  • [MEDIUM/cognitive_load] Oversized files x4 – benchmarks/src/datasets.ts, packages/toon/src/decode/decoders.ts, docs/.vitepress/theme/components/PlaygroundLayout.vue (606 lines each). Fix: split by responsibility.
  • [MEDIUM/cognitive_load] High branching density x6 – packages/toon/src/index.ts, packages/toon/src/encode/normalize.ts, benchmarks/src/formatters.ts (25 branch points over 83 lines). Fix: decompose decision-heavy logic.
  • [MEDIUM/clarity] Hub modules x5 – benchmarks/src/types.ts, packages/toon/src/types.ts, packages/toon/src/constants.ts (17 dependents each). Fix: keep stable and small; move volatile logic out.

SDLC observations: the repo has tests, CI (GitHub Actions), a license (MIT), and a lockfile (pnpm-lock.yaml), but no Dockerfile. No committed secrets were found. The measured findings are deterministic outputs of this pipeline; no additional issues should be appended beyond what the block reports.

The Bottom Line

TOON delivers a real token-reduction win for LLM prompts involving uniform object arrays, with a working TypeScript SDK and CLI. The codebase is structurally sound but has cognitive-load hotspots—oversized files, high branching density, and duplicated logic—that should be addressed before wide adoption. It’s worth using when your data fits TOON’s format and you’re token-constrained; otherwise, stick with JSON.