Here's a concise, professional technical briefing for the privacy-filter.cpp repository, written in the style of a senior AI engineer consultant report.

privacy-filter.cpp

The Problem

This repository delivers a minimal GGML-based inference engine for OpenAI’s privacy-filter NER model family, enabling PII entity span detection with exact UTF-8 byte offsets. It is not a general-purpose LLM framework — it is a focused, performance-conscious implementation that trades breadth for speed and memory efficiency. The core value proposition is real-time redaction on long documents where stock HuggingFace Transformers OOM or collapse under O(n²) self-attention.

What This Does

The project is a portfolio of 6 self-contained subprojects: src, scripts, bench, demo, fuzz, and tools. The substantial ones are:

  • src/model.cpp and src/ner.cpp — the GGML model loader and NER inference pipeline. These files implement the forward pass, RoPE frequency factor computation, and entity span extraction. Indentation depth reaches 11; control flow is dense and would benefit from guard-clause flattening.
  • scripts/convert.py — a self-contained checkpoint-to-GGUF converter with no llama.cpp dependency.
  • bench/pf-bench.cpp — microbenchmarks comparing privacy-filter throughput against HuggingFace Transformers on CPU and GPU.
  • demo/pii_duel.py and demo/gen_corpus.py — demo scripts that drive the CPU/GPU redaction race videos committed in demo/out/.

The src/ directory contains 6 code files (backend.cpp, gguf_loader.cpp, model.cpp, ner.cpp, pf.cpp, tokenizer.cpp) and 6 headers. Execution flows from pf.cpp (entry point) → backend.cpp (GGML dispatch) → model.cpp (RoPE + attention) → ner.cpp (span extraction). The gguf_loader.cpp/h pair handles GGUF model deserialization. No circular import edges exist across 35 analyzed files.

How It Is Wired

Execution starts at pf.cpp (the CLI entry point), which parses arguments and dispatches to backend.cpp. The backend initializes the GGML context, loads the GGUF model via gguf_loader.cpp, and routes the input through model.cpp where RoPE frequency factors are applied at load time and the banded attention forward pass executes. ner.cpp then extracts entity spans with byte offsets. Output is printed to stdout or written to a file depending on the invoked mode.

The scripts/convert.py file is the only path from a HuggingFace checkpoint to a runnable GGUF model — it downloads the HF checkpoint, extracts config, and writes a .gguf binary. No llama.cpp build step is required.

Outside the repo, the engine reads a .gguf model file and writes plaintext to stdout; it does not phone home, persist state, or require network access during inference.

How To Use It

Build — The repo uses CMake with presets. A full build with CPU and Vulkan GPU support:

git clone --recursive https://github.com/moses-y/privacy-filter.cpp
cmake --preset release-portable && cmake --build --preset release-portable -j

GPU backends layer onto any preset via -DPF_VULKAN=ON or -DPF_CUDA=ON. CMakePresets.json documents the available presets (release, debug, profile, fuzz).

Convert a checkpoint — run the self-contained converter:

python scripts/convert.py <hf-checkpoint> <output.gguf>

Run inference — the compiled binary is build/release-portable/bin/pf-cli:

build/release-portable/bin/pf-cli --info model.gguf

See README.md for the full command reference, including benchmark invocation.

Real-World Use

A privacy-conscious LLM deployment can embed this engine as a preprocessing step before any downstream generation. Given a document, it returns a list of (start_byte, end_byte, entity_type) tuples with exact UTF-8 offsets, enabling redaction without touching the model weights. Because the model runs in GGML, it can be embedded in a C++ service, a Rust FFI boundary, or invoked via the CLI from a shell pipeline. The banded attention design means throughput stays flat as document length grows, unlike full-self-attention Transformers that OOM past a few thousand tokens.

Code Health & Issues

The static analysis (35/35 files) found 21 issues across 4 kinds:

  • Deep nesting x12 in src/model.cpp, src/ner.cpp, src/ner.h — control flow is hard to follow at max indentation depth 11. Fix: flatten with early returns/guard clauses.
  • Duplicated code blocks x17 across tests/test_graph_blocks.cpp, tests/test_parity.cpp, tests/test_tokenizer.cpp, tests/test_window_stitch.cpp — 6-line blocks repeated 5 files. Fix: extract shared helpers.
  • File opened without context manager x2 in demo/gen_corpus.py, demo/pii_duel.py — open(...) not wrapped in with. Fix: use with open(...) as f:.
  • High branching density x6 in fuzz/fuzz_tokenizer.cpp, src/backend.cpp, src/ner.cpp — 22 branch points over 48 lines. Fix: decompose decision-heavy logic.

SDLC observations from the file structure:

  • No Dockerfile present; builds are host-native only.
  • No lockfile for scripts/requirements.txt — non-reproducible installs.
  • GitHub Actions CI present but lacks dependency vulnerability scanning.
  • Large media blobs (GIFs/MP4s over 5MB) committed to the repo — every clone pays for data not diffed.
  • 6 generated build outputs committed under demo/ — stale bundles possible.
  • No license boundary declared in CI (GITHUB_TOKEN inherits repo default).
  • No Dependabot/Renovate configuration.
  • No persist-credentials: false on checkout step.

The Bottom Line

This is a well-scoped, performance-motivated implementation that delivers on its promise: a GGML-based NER engine that runs circles around HuggingFace Transformers on long-document redaction. The code health is fair — nesting is deep, tests have duplicated logic, and the CI hygiene is thin. If you need a lightweight, embeddable PII filter that scales to 100k+ token documents without GPU memory explosion, this is the right tool. If you want a polished framework with lockfile-managed dependencies, CI gates, and structured docs, look elsewhere.