The Problem

Extracting structured knowledge from unstructured text usually means heavy NLP pipelines, NER models, and graph databases. This repo offers a cheaper path: use a local LLM (Mistral 7B via Ollama) to extract concepts and relationships from text chunks, then assemble a queryable knowledge graph for Graph-Augmented Generation or Q&A.

What This Does

The pipeline splits a corpus into chunks, asks an LLM to extract concepts and relations per chunk, then builds a graph where edges represent either explicit LLM-derived relations or contextual proximity between concepts in the same chunk. The heavy lifting lives in three modules: helpers/prompts.py (LLM prompts), helpers/df_helpers.py (DataFrame transformations), and ollama/client.py (a thin Ollama API wrapper).

The main workflow is extract_graph.ipynb. Supporting notebooks in old_notebooks/ show earlier iterations. Sample data lives in data_input/ (PDFs, text) and data_output/ (generated CSVs).

How It Is Wired

Execution starts in extract_graph.ipynb, which calls into the helper modules. The call graph is small: 4 internal modules, 2 import edges, 17 resolved call edges, no cycles. The three most-called functions are generate (called from 3 places), graphPrompt (2), and extractConcepts (2) — these are the blast-radius points; changing their signatures breaks most of the pipeline.

The data flow is: documents2Dataframe → df2ConceptsList → extractConcepts → generate (network call to Ollama) → concepts2Df → df2Graph → graphPrompt → generate → graph2Df. Nine functions make outbound network calls — all through ollama/client.py. One function reads/writes files.

The wiring is simple and changeable: helpers/df_helpers.py is pure DataFrame logic with no external calls, so it's easy to test in isolation. ollama/client.py is the only network boundary and carries the most risk: deep nesting (indentation depth 7) and no timeouts on any of its 9 outbound calls.

How To Use It

git clone https://github.com/moses-y/knowledge_graph
cd knowledge_graph
poetry install
ollama pull mistral:7b-openorca

Open extract_graph.ipynb and run it. The notebook expects an Ollama server running locally; no API keys are required. The README documents this as a no-GPT, fully local setup. The dockerfile and environment.yml provide container/conda alternatives, but the notebook is the real entry point.

Real-World Use

For a document corpus (e.g., medical literature, as in data_input/cureus/), you'd run the notebook to produce graph.csv, then load it into a graph database or use it as a retriever for RAG. The output CSVs in data_output/ are the contract — chunks, concepts, and edges — so downstream systems can consume them without touching the extraction code.

Code Health & Issues

Static analysis found 2 medium findings. helpers/prompts.py uses broad exception handling that swallows errors indiscriminately — catch specific exceptions instead. ollama/client.py has deeply nested control flow (depth 7) and 9 outbound calls with no timeout, so a stalled Ollama peer can hold a worker indefinitely — pass timeout= on every call.

SDLC gaps: no tests, no CI, no license file, no Dependabot/Renovate config. Dependencies are stale — pypdf is 3 majors behind, langchain 1 major behind. A 19.4MB PDF in assets/ bloats every clone; move it to LFS or object storage.

The Bottom Line

A practical, working demonstration of LLM-based knowledge graph extraction with a clean three-module core and no external API costs. It's a notebook-driven prototype, not a library — no tests, no CI, no packaging for reuse. Fork it for experimentation or as a reference implementation, but productionize the Ollama client and add tests before relying on it.