The Problem

Detecting a file’s true content type quickly and reliably is still a pain point for security pipelines, data‑ingestion services, and any system that must route files to the appropriate handler. Conventional magic‑byte sniffers miss many edge cases and often require heavyweight libraries, leading to latency spikes or false classifications.

What This Does

magika provides an AI‑driven classifier that runs inference in ~5 ms per file on a single CPU while keeping the model under a few megabytes. The repository bundles seven self‑contained projects:

  • Python API – python/src/magika/ (e.g. magika.py, types/) exposes Magika, identify_path, identify_bytes, etc.
  • Rust CLI – rust/cli/src/main.rs implements the command‑line tool.
  • Go bindings – go/cli/main.go and go/example/main.go wrap the same core logic.
  • JavaScript/TypeScript – js/ contains the npm package (magika, magika-node) and several simple examples.
  • Web demo – website-ng/ and website/ host the browser‑based UI.
  • Model assets – assets/models/ hold the ONNX/TensorFlow/Keras binaries used at inference time.
  • Test data – tests_data/ supplies the 185 files used by the test suites.

Key shared code lives in js/magika* (the most imported JS module) and python/src/magika/magika.py (the hub for the Python surface).

How It Is Wired

Execution starts at the language‑specific entry point:

  • Go CLI – go/cli/main.go → calls Magika (the Go wrapper) → invokes identify_path/identify_bytes.
  • Rust CLI – rust/cli/src/main.rs → loads the model → calls identify_stream.
  • Node/Browser – js/simple_examples/.../index.js → imports js/magika → uses Magika → ultimately reaches identify_path.

Across languages the core flow is identical:

  1. Model load – python/src/magika/magika.py::_load_model_config creates a ContentTypeLabel (4 calls).
  2. File identification – public functions identify_path, identify_bytes, identify_stream, identify_paths are called from the CLI or API (31, 15, 9, 8 callers respectively).
  3. Result construction – MagikaResult (in python/src/magika/types/magika_result.py) aggregates the label, confidence, and status.

The internal call graph shows Magika as the highest‑fan‑out node (31 distinct callers). Functions such as get_basic_test_files_paths and _get_default_model_name are also widely used (≥6 callers). No circular import cycles were detected, simplifying static reasoning.

External interactions are limited to three categories:

  • Subprocess – python/scripts/check_copyright.py → subprocess.run (via get_relevant_files_paths).
  • File I/O – 28 functions read/write files, e.g., model loading and test‑data discovery.
  • Network – python/scripts/check_documentation.py performs an outbound HTTP request to fetch the latest Python version.

Thus a typical run touches the model file, reads the target file(s), and returns a JSON‑compatible result without side‑effects beyond optional logging.

How To Use It

# Clone the repository
git clone https://github.com/moses-y/magika
cd magika

# Python package (editable install)
pip install -e python

# JavaScript/TypeScript package
cd js
yarn install

# Rust CLI
cd rust/cli
cargo build --release

# Go wrapper (requires Go 1.22+)
cd go/cli
go build -o magika

Running the CLI examples:

# Go
./magika -r path/to/dir

# Rust (released binary)
./target/release/magika path/to/file

# Node
node js/simple_examples/node-commonjs-example/index.js path/to/file

The model files are expected under assets/models/; no additional configuration is required.

Real‑World Use

A security‑scanning pipeline can invoke magika as a pre‑filter:

steps:
  - name: Identify file types
    run: |
      ./magika -r $INPUT_DIR > identified.json
  - name: Route to scanners
    uses: ./scan_based_on_type.sh identified.json

The fast, constant‑time inference lets the pipeline process thousands of files per minute while keeping the CPU footprint low.

Code Health & Issues

Measured static findings (20 total)

  • HIGH – Deep nesting – rust/cli/src/main.rs, python/src/magika/cli/magika_client.py, go/magika/scanner_test.go (max depth 8).
  • HIGH – Duplicated code – repeated 6‑line blocks across js/magika-cli.ts, js/magika-node.ts, js/magika.ts, js/src/content-type-info.ts (73 occurrences).
  • HIGH – Oversized files – js/src/content-types-infos.ts, rust/lib/src/content.rs, rust/lib/src/model.rs (≈1.4 k LOC each).
  • MEDIUM – Empty catch block – tests_data/basic/javascript/code.js.
  • MEDIUM – Broad exception handling – python/scripts/pre_release_check.py (bare except).
  • MEDIUM – High branching density – rust/gen/src/main.rs, website-ng/src/lib/components/ui/file-drop-zone/file-drop-zone.svelte, tests_data/basic/javascript/code.js (107 branches / 209 lines).

Code‑health audit (7 findings)

  • HIGH – Pin GitHub Actions to commit SHAs (.github/workflows/*).
  • MEDIUM – Pin Docker base image by digest (Dockerfile).
  • MEDIUM – Add a dependency‑vulnerability scan to CI.
  • MEDIUM – Move large model binaries (>5 MiB) to Git LFS (assets/models/standard_v2_1/model.keras).
  • MEDIUM – Set persist-credentials: false on checkout (python-build-and-release-package.yml).
  • MEDIUM – Add a non‑root USER in Dockerfile.
  • LOW – Define timeout-minutes on workflow jobs (cli-release.yml).

All findings stem from deterministic static analysis; no additional issues were observed.

The Bottom Line

magika delivers a fast, AI‑based file‑type detector across multiple languages with a clean, well‑instrumented code base. The primary engineering concerns are high cognitive load in several large or deeply nested files and a few hygiene gaps (unpinned actions, oversized binaries). Teams needing high‑throughput, low‑latency content classification can adopt the library, provided they address the listed health items before production rollout.