OpenKB – Technical Briefing

What the problem is Enterprises that ingest long‑form documents (PDFs, slides, spreadsheets) often rely on ad‑hoc retrieval: each query re‑processes the raw files, so knowledge never accumulates and cross‑references are missing. Maintaining a persistent, searchable wiki that can be updated incrementally while still supporting multi‑modal content is the gap OpenKB addresses.

What OpenKB does OpenKB is a Python CLI that compiles arbitrary files into a plain‑text markdown wiki (.md with [[wikilinks]]). It uses PageIndex’s tree‑indexing for long PDFs, feeds the extracted text to an LLM via LiteLLM, and writes inter‑linked concept pages. The wiki can be queried (openkb query) or chatted with (openkb chat) in a multi‑turn session that persists across runs. A built‑in linter (openkb lint) flags broken links, contradictions, and orphaned pages. The output is Obsidian‑compatible, letting users browse the graph visually.

How it is wired

  1. Entry point – openkb/cli.py defines the sub‑commands. The init command (line 257) creates the KB directory and calls load_config (via save_config). add (line 325) parses added paths, then invokes _persist in openkb/state.py, which writes files under the KB folder (self._path.parent.mkdir).
  2. Core flow – Adding a document triggers openkb/agent/compiler.py.start, which calls _run → _make_wiki (≈18 call sites). _make_wiki builds the markdown tree, then _update_index in openkb/indexer.py updates the PageIndex structure. The most‑called internal helper is _setup_kb (30 callers).
  3. LLM interaction – openkb/agent/chat.py.run_chat eventually calls _fmt → _build_style → the LLM via litellm (the only function that performs inference).
  4. Filesystem touch – Over 165 functions read/write files: save_config, load_config, HashRegistry in openkb/state.py (cryptographic hash generation), and the watcher (openkb/watcher.py) which reacts to new files via on_new_files.
  5. Cycles & complexity – openkb/cli.py, openkb/agent/chat.py, and openkb/agent/query.py form a circular import (3 modules). Each contains deep nesting (max indentation depth 9) and duplicated 6‑line blocks for argument parsing and output formatting. The compiler (openkb/agent/compiler.py) and its test (tests/test_compiler.py) are oversized (≈629 lines each), making local changes high‑risk.

How to use it

# Install from PyPI
pip install openkb

# Or from the repo (editable)
git clone https://github.com/moses-y/OpenKB
cd OpenKB
pip install -e .
# Initialise a new knowledge base
mkdir my-kb && cd my-kb
openkb init          # creates .openkb/config.yaml and raw/ folder

# Add documents (PDF, DOCX, MD, etc.)
openkb add paper.pdf
openkb add ./reports/   # bulk add a directory

# Query the compiled wiki
openkb query "What are the main findings?"

# Interactive chat (session persisted)
openkb chat

Environment: create a .env with LLM_API_KEY=… as described in the README. No other secret files are required.

Real‑world scenario A research team stores all conference papers in my-kb/raw/. Running openkb add *.pdf builds a linked markdown wiki where each paper’s concepts are cross‑referenced automatically. When a new manuscript arrives, the watcher (openkb/watcher.py) detects the file, re‑runs the compiler, and updates the wiki without manual intervention. Analysts can then ask the LLM for synthesis across years of papers via openkb query.

Code health & issues (static‑analysis findings)

  • High – No lockfile (pyproject.toml only). Commit a lockfile to guarantee reproducible builds.
  • High – CI workflow (.github/workflows/publish.yml) never runs the test suite. Add a pytest step.
  • Medium – Import cycle among cli.py, chat.py, query.py. Break the cycle by extracting shared types or using lazy imports.
  • Medium – Deep nesting (max depth 9) in the same three modules; refactor with early returns or helper functions.
  • Medium – Duplicated code blocks across chat.py, query.py, cli.py, linter.py; extract to a common utility.
  • Medium – Oversized files (agent/compiler.py, tests/test_compiler.py > 600 lines); split by responsibility.
  • Medium – Files opened without context manager (openkb/converter.py, openkb/images.py). Replace with with open(...) as f:.
  • Medium – Broad except: clauses in openkb/images.py; catch specific exceptions.
  • Medium – No Dependabot/renovate bot; add .github/dependabot.yml.
  • Medium – No dependency‑vulnerability scan in CI; integrate dependency-review-action or osv-scanner.
  • Medium – Large PDFs (≈17 MB) stored in the repo; move to Git LFS or external storage.
  • Medium – Checkout step persists the token; set persist-credentials: false.
  • Low – CI jobs lack timeout-minutes; define a reasonable timeout.

Bottom line OpenKB delivers a functional, LLM‑driven wiki pipeline with concrete CLI commands and a decent test suite, but the codebase suffers from import cycles, deep nesting, and missing production safeguards (lockfile, proper CI testing, dependency scanning). It is suitable for teams that need rapid prototyping of document‑centric knowledge bases and are willing to invest in refactoring and CI hardening before production use.