llmwiki

The Problem

Users need a personal knowledge base that stays up‑to‑date without manual editing. Existing wiki tools require manual copy‑pasting, and integrating large‑language‑model (LLM) assistants is ad‑hoc, leading to stale or incomplete notes.

What This Does

llmwiki implements Karpathy’s “LLM Wiki” concept as a self‑maintaining second brain. It ingests PDFs, Word files, PowerPoints, Markdown, and web clippings (via a Chrome extension), stores them in a SQLite‑backed workspace, and exposes a Next.js UI (web/). A Claude‑compatible MCP server (mcp/) lets Claude read, write, and search the indexed content, enabling scheduled “Claude Routines” that synthesize new wiki pages nightly. The repo is a collection of seven loosely coupled projects (API, web UI, browser extension, MCP server, Supabase helpers, document converter, shared utilities) rather than a monolith.

How It Is Wired

Primary entry points

Typical request flow (e.g., GET /documents)

1. HTTP server (api/main.py) creates a DB pool (create_pool).
2. Router api/routes/documents.py calls list_documents → api/services/base.py.
3. Service uses auth headers (auth_headers, called from 90 places) to verify the user.
4. Data is fetched via fetchrow/fetch (83 and 31 callers) in api/scoped_db.py, which wraps SQLite (api/infra/db/sqlite.py).
5. Results are serialized and returned to the Next.js front‑end (web/).

Key hub modules

• api/infra/db/sqlite.py – 51 functions, 5 classes, read/write DB & files; called from 21 files.
• tests/unit/test_html_parser_highlights.py – high test coverage of the HTML parser.
• api/services/hosted.py – 55 functions handling user profile and usage tracking.

Outbound effects

• Database I/O: 159 functions (mostly via sqlite.py).
• Network calls: 114 functions (e.g., Claude MCP, external OCR service).
• File system: 51 functions (workspace creation, PDF/Word conversion).

The call graph shows no circular imports and a shallow dependency depth, but several hub functions (auth_headers, fetchrow, create_document, _make_kb) have the widest blast radius—changing them can affect dozens of callers.

How To Use It

# Clone the original repository (forked from lucasastorian/llmwiki)
git clone https://github.com/moses-y/llmwiki.git
cd llmwiki

# Python environment (requires Python 3.11+)
python -m venv .venv && source .venv/bin/activate
pip install -r api/requirements.txt -r mcp/requirements.txt

# Node.js UI
cd web && npm install && cd ..

# Start the stack (Docker optional, see docker-compose.yml)
./llmwiki open ~/research   # initializes workspace, indexes files, launches API + UI

Configuration – copy .env.example to .env and set WORKSPACE_ROOT (the folder you want indexed). The MCP server reads mcp/config.py; after running ./llmwiki mcp-config ~/research paste the printed JSON into Claude Desktop’s claude_desktop_config.json (or .claude/settings.json).

Running the API directly – uvicorn api.main:app --host 0.0.0.0 --port 8000.

Extension – load extension/ as a Chrome unpacked extension; it uses background/index.ts and content/index.ts to capture highlights and upload via the API.

Real‑World Use

A researcher stores all PDFs and notes in ~/papers. After the initial llmwiki open run, the workspace is searchable in the web UI. The researcher adds a Claude Routine that, each night, calls the MCP endpoint POST /kb/{kb_id}/ingest (implemented in api/routes/knowledge_bases.py). Claude reads newly added PDFs, extracts highlights via api/services/ocr.py, and creates a synthesized wiki page via create_document → api/services/local.py. The result appears instantly in the UI and is cross‑linked to source documents.

Code Health & Issues

• High – Pin GitHub Action versions to commit SHAs (.github/workflows/*).
• Medium – Declare least‑privilege GITHUB_TOKEN permissions (test.yml).
• Medium – Enable Dependabot (.github/dependabot.yml).
• Medium – Pin Docker base image by digest (api/Dockerfile).
• Medium – Set persist-credentials: false on checkout (security.yml).
• Medium – Add non‑root USER in Dockerfile.
• Low – Add timeout-minutes to workflow jobs.
• Low – Provide repo‑wide convention files (.editorconfig, formatter config).

Static analysis also reports:

• High cognitive load: deep nesting in mcp/tools/references.py and api/services/ocr.py.
• High clarity: duplicated 6‑line blocks across auth modules (api/auth.py, mcp/auth.py).
• Medium resilience: broad except: clauses in parsers (api/services/parsers.py).
• Medium cognitive load: oversized files (api/html_parser/parser.py ~ 978 lines).

No critical findings, but the highlighted hotspots increase maintenance effort and should be prioritized for refactoring.

The Bottom Line

llmwiki delivers a functional, containerizable LLM‑augmented wiki with a clear separation of concerns across API, UI, and MCP components. It is usable out‑of‑the‑box, but the codebase contains several high‑impact readability and duplication issues, and its CI/CD security posture needs tightening. It is suitable for teams comfortable with Python/TypeScript stacks that want an extensible personal knowledge base and are prepared to address the identified technical debt.