The Problem

Users who consume content across many platforms (GitHub, HackerNews, Zotero, HuggingFace, etc.) end up with a scattered set of bookmarks that are hard to search later. Maintaining a personal knowledge base that stays up‑to‑date and offers fast, semantic lookup requires custom crawling, indexing, and a UI, which most developers would have to build from scratch.

What This Does

knowledge automates the whole pipeline. A daily GitHub Actions workflow pulls data from the supported services, stores the raw records in database/database.json, builds a topic graph in knowledge_database/graph/graph.py, and serialises the search pipeline to database/pipeline.pkl. The static front‑end (docs/index.html) loads a WebAssembly version of the pylate-rs model (bundled in docs/pkg/) to run queries entirely in the browser. The backend API lives in api/api.py and is launched by run.py (FastAPI) inside the Docker container defined by Dockerfile.

Key files:

  • knowledge_database/pipeline/pipeline.py – orchestrates loading the graph, indexing, and exposing search and plot.
  • knowledge_database/graph/graph.py – builds the knowledge graph, provides __call__ for traversal and yens for shortest‑path queries.
  • api/api.py – FastAPI routes that call pipeline.get_latest_documents, pipeline.search, and pipeline.plot.
  • docs/pkg/pylate_rs.* – compiled Rust/WASM model used by the browser UI for lexical similarity.

How It Is Wired

Execution starts with the Docker entrypoint defined in Dockerfile which runs python run.py. run.py creates a FastAPI app and mounts the routes from api/api.py.

  1. API request → api/api.py → calls pipeline.search (or pipeline.get_latest_documents).
  2. pipeline.search (in knowledge_database/pipeline/pipeline.py) loads the serialized graph (database/pipeline.pkl) and forwards the query to knowledge_database/graph/graph.py.__call__.
  3. graph.__call__ performs a lexical lookup using the pylate-rs model (imported via the generated JS/WASM bundle when the front‑end runs) and returns matching document IDs.
  4. Results travel back through pipeline.search → API response → front‑end JavaScript, which renders them in the UI.

The most‑connected module is knowledge_database/pipeline/pipeline.py (imports 2 modules, imported by 1). It has an instability of 0.67, meaning changes there affect downstream code (api/api.py) but it does not depend on many others. The graph module (knowledge_database/graph/__init__) is also a hub with equal inbound/outbound counts (instability 0.5). No circular imports were detected, so the call chain is linear and easy to trace.

How To Use It

# 1. Clone the repo (use the exact URL)
git clone https://github.com/moses-y/knowledge
cd knowledge

# 2. Build the container
docker build -t knowledge .

# 3. Run the service (exposes port 8000 by default)
docker run -p 8000:8000 knowledge

Python dependencies are declared in pyproject.toml; the lockfile uv.lock is present, so install locally with uv sync if you prefer a native environment.

Front‑end assets are static; open docs/index.html in a browser after the container is running (the UI fetches data from http://localhost:8000).

Secrets (API keys for Fly.io, Zotero, etc.) must be added as repository secrets for the GitHub Actions workflows (.github/workflows/*.yml). The back‑end reads them via the standard os.getenv pattern (see run.py for the FastAPI start‑up).

Real‑World Use

A developer who wants a searchable “second brain” can fork this repo, supply their own GitHub token, Zotero key, and Fly.io token, and let the nightly workflow keep the graph current. An internal tool could then query GET /search?q=transformer against the FastAPI endpoint, receiving a JSON payload of matching bookmarks without any additional indexing infrastructure.

Code Health & Issues

  • Medium – resilience – Broad except: blocks in knowledge_database/graph/graph.py and knowledge_database/huggingface/huggingface.py swallow errors.
  • Medium – cognitive_load – Deep nesting (6 levels) in knowledge_database/graph/graph.py makes the logic hard to follow.
  • Medium – clarity – Nearly identical 6‑line snippets appear in api/api.py, knowledge_database/pipeline/pipeline.py, knowledge_database/hackernews/hackernews.py, and knowledge_database/huggingface/huggingface.py; extracting a shared helper would reduce duplication.
  • Low – hygiene – No lockfile for docs/pkg/package.json; builds that depend on npm may be non‑reproducible.
  • Low – security – Image img/secrets.png suggests a secret‑shaped path was committed; audit required.

The repo includes a single test file (pytest.ini indicates pytest usage) and CI pipelines (.github/workflows/*.yml). License is present, but the committed secret‑shaped image is a red flag.

The Bottom Line

knowledge delivers a functional, end‑to‑end personal bookmark search engine with a clear separation between data ingestion, graph construction, and UI rendering. It is usable out‑of‑the box for developers comfortable with Docker and FastAPI, but the codebase suffers from generic exception handling, deep nesting, and duplicated snippets that will increase maintenance effort. Ideal for hobbyists or small teams needing a self‑hosted knowledge graph; less suited for production‑grade deployments without refactoring the highlighted hotspots.