sift kg

Abstract

Turn any collection of documents into a knowledge graph. Extract entities and relationships via LLM, deduplicate with your approval. Map domains, find hidden connections, spot patterns across documents — knowledge that persists and compounds, for you and your AI agents. All from the CLI.

The Problem

Structured knowledge extraction from document collections is manual, error-prone, and doesn't scale. Teams resort to spreadsheets or ad-hoc scripts, losing the relationships between entities and the provenance linking each fact to its source. The result is knowledge that doesn't compound—it sits in silos, undocumented and unreusable.

What This Does

sift-kg turns a folder of documents into a queryable knowledge graph via a CLI. It extracts entities and relationships using an LLM, deduplicates them with human approval, and generates an interactive browser viewer. The graph persists as JSON and exports to GraphML, GEXF, SQLite, or CSV.

The pipeline is orchestrated by src/sift_kg/cli.py and src/sift_kg/pipeline.py. Extraction logic lives in src/sift_kg/extract/, entity resolution in src/sift_kg/resolve/, and graph construction in src/sift_kg/graph/. The viewer is a static HTML/JS app in src/sift_kg/viewer/ with no server component.

How It Is Wired

Execution starts at src/sift_kg/__main__.py, which delegates to cli.py. The CLI commands map directly to pipeline stages: extract → extractor.py, build → graph/builder.py, resolve → resolve/engine.py, review → resolve/reviewer.py, narrate → narrate/generator.py, view → opens the static viewer.

The pipeline is linear: documents → text extraction (ingest/) → schema discovery (domains/) → entity/relation extraction (extract/) → graph construction (graph/) → deduplication (resolve/) → narrative generation (narrate/). Each stage writes intermediate JSON to disk, so stages can be re-run independently.

The widest blast radius is src/sift_kg/cli.py at 1,239 lines—it handles all command routing, configuration, and orchestration. src/sift_kg/viewer/app.js (1,203 lines) carries the entire interactive viewer with 418 branch points. Both are oversized and hard to modify safely.

The module graph shows 57 internal modules with zero import edges detected, meaning the static analysis couldn't resolve the import structure. The wiring between modules is real but not mapped by this analysis.

How To Use It

pip install sift-kg

sift init                           # create sift.yaml + .env.example
sift extract ./documents/           # extract entities & relations
sift build                          # build knowledge graph
sift resolve                        # find duplicate entities
sift review                         # approve/reject merges interactively
sift apply-merges                   # apply your decisions
sift narrate                        # generate narrative summary
sift view                           # interactive graph in your browser
sift export graphml                 # export to Gephi, yEd, Cytoscape, SQLite, etc.

Configuration lives in sift.yaml and .env.example at the repo root. LLM provider keys (OpenAI, Anthropic, Mistral, or local Ollama) go in the environment file. The examples/ directory shows complete workflows—examples/ftx/ is a full run with outputs.

Real-World Use

An investigator analyzing a document dump—say, court filings or leaked communications—runs sift extract on a folder of PDFs, reviews proposed entity merges with sift review, then exports the graph to GraphML for analysis in Gephi. The narrative output (narrate/generator.py) produces a prose summary with relationship chains, useful for briefings.

Code Health & Issues

Static analysis found 18 issues: 2 high, 16 medium. The high-severity findings are oversized files—src/sift_kg/cli.py (1,239 lines), src/sift_kg/viewer/app.js (1,203 lines), and src/sift_kg/narrate/generator.py—all hard to hold in one head. Medium findings include files opened without context managers (ingest/ocr.py, visualize.py), duplicated code blocks across 8 files, broad exception handling in graph/knowledge_graph.py and narrate/generator.py, and deep nesting in cli.py and llm_client.py.

The code health audit flags one high issue: no lockfile beside pyproject.toml, so builds aren't reproducible. Four medium issues: no least-privilege permissions on the GitHub Actions token, no Dependabot config, no dependency vulnerability scan in CI, and persist-credentials: false not set on checkout. One low issue: no job timeouts in the workflows. Tests exist (16 files) and CI runs via GitHub Actions.

The Bottom Line

sift-kg is a functional, well-scoped tool for document-to-knowledge-graph pipelines with a sensible human-in-the-loop design. The main risks are maintainability—oversized files and duplicated logic—and supply-chain hygiene around the missing lockfile. Teams needing structured extraction from unstructured documents will find it useful; those wanting a clean codebase to extend should budget for refactoring first.