The Problem

Extracting structured data from PDFs and images typically requires OCR pipelines that fail on complex layouts, handwriting, and embedded tables. Most solutions are cloud-hosted, forcing sensitive documents off-premises. Teams need a self-hosted way to convert documents to markdown and evaluate VLM performance on document tasks without sending data to third parties.

What This Does

docext is an on-premises toolkit for document intelligence. It converts PDFs and images to structured markdown with semantic tags for signatures, watermarks, equations, and tables, and extracts structured fields with confidence scores. It also includes a benchmarking harness that evaluates VLM performance across seven document tasks: KIE, VQA, OCR, classification, table extraction, and more.

The core logic lives in docext/core/ — extract.py handles field extraction, pdf2md/pdf2md.py manages markdown conversion, and confidence.py scores outputs. The benchmark system in docext/benchmark/ contains dataset loaders (vlm_datasets/), task definitions (tasks.py), and metrics (metrics/).

How It Is Wired

Execution starts at docext/app/app.py, a Flask server exposing the API. It routes to pdf2md.py for conversion and extract.py for field extraction. The benchmark entry point is docext/benchmark/benchmark.py, which orchestrates dataset loading, model inference, and metric computation.

The module graph shows docext/benchmark/vlm_datasets/ds.py as a hub: 16 modules depend on it, making it the highest-blast-radius file. benchmark.py (635 lines) and tasks.py are the main orchestration layers. The system calls out to VLM inference via docext/core/vllm.py and the client.py module — these are the network boundaries.

The ds.py hub is a concern: any change there ripples through 16 dependents. The duplicated 6-line blocks across 12 files (103 occurrences) suggest extraction logic was copy-pasted rather than shared.

How To Use It

Setup: Install via pip from requirements.txt. A Dockerfile exists for containerized deployment.

pip install -r requirements.txt
# or
docker build -t docext .

Configuration: The configs/benchmark.yaml file controls benchmark settings. No environment variables are documented in the repo.

Running: Start the Flask app from docext/app/app.py:

python docext/app/app.py

The repo also includes Jupyter notebooks (docext.ipynb, pdf2markdown.ipynb) for interactive use.

Real-World Use

A team handling invoices processes them locally. They run the Flask server, POST a PDF to the conversion endpoint, receive markdown with <signature> and <table> tags, then use the extraction endpoint to pull structured fields (invoice number, total, vendor). The benchmark harness lets them compare a self-hosted VLM against published leaderboard scores before committing to it.

Code Health & Issues

Static analysis found 17 findings: 3 high, 14 medium. The high-severity issues are:

  • High — Deep nesting (max depth 6) in ds.py, pdf2md.py, and benchmark.py; control flow is hard to follow.
  • High — 103 duplicated 6-line blocks across 12 files, including app.py, pdf2md.py, chartqa.py, and docvqa.py.
  • Medium — ds.py is a hub module with 16 dependents; changes there have wide blast radius.
  • Medium — Unmanaged file handles in ds.py; open() without context manager.
  • Medium — Broad exception handling in metrics/grits.py swallows errors.

The repo has no test suite despite 43 Python source files. CI exists (GitHub Actions) but only runs CodeQL — no dependency vulnerability scanning. The Docker base image (vllm/vllm-openai:v0.8.2) is unpinned by digest, and no lockfile means non-reproducible builds. The gradio and transformers dependencies are one major version behind current releases.

The Bottom Line

docext is a functional toolkit for on-premises document extraction and VLM benchmarking, with a sensible module split between core extraction and benchmark harnesses. The lack of tests and lockfile, plus heavy duplication in the dataset loaders, make it a maintenance risk for anyone planning to extend it. It is a reasonable starting point for teams that need self-hosted document intelligence, but expect to invest in test coverage and dependency hygiene before production use.