The Problem

Fine‑tuning large language models (LLMs) with high‑quality, domain‑specific question‑answer data is costly and time‑consuming. Existing pipelines either rely on manual annotation or generate synthetic QA without grounding in a structured knowledge source, leading to low coverage of long‑tail facts and sub‑optimal downstream performance.

What This Does

GraphGen is a Python‑based framework that builds a fine‑grained knowledge graph from raw documents, detects knowledge gaps in a target LLM, and synthesises QA pairs that target those gaps.

Core components – the graphgen/ package defines a clean abstraction hierarchy: bases/basekgbuilder.py, bases/basegenerator.py, bases/baseextractor.py provide the contract for KG construction, data generation and information extraction. Concrete implementations live under graphgen/models/, e.g. kgbuilder/lightragkgbuilder.py (RAG‑style KG), generator/atomicgenerator.py (single‑fact QA), generator/multihopgenerator.py (multi‑hop reasoning), and llm/api/openaiclient.py / ollamaclient.py for LLM access.

Engine – graphgen/engine.py orchestrates the workflow: it loads configuration, initializes the LLM wrapper (graphgen/common/initllm.py), builds the KG, runs the gap detector, and dispatches the appropriate generator.

Examples – the examples/ directory ships end‑to‑end scripts (e.g. examples/generate/generateatomicqa/generateatomic.sh) that invoke the engine with ready‑made config YAML files such as atomicconfig.yaml. These scripts illustrate typical use‑cases: QA synthesis, multi‑hop generation, VQA, and evaluation pipelines (examples/evaluate/...).

How To Use It

Setup # Clone and enter the repo git clone https://github.com/open-sciencelab/GraphGen.git cd GraphGen

Install Python dependencies (the Dockerfile lists them; replicate locally)

pip install -r requirements.txt # if a requirements.txt is added; otherwise: pip install -e . # installs the package in editable mode

Container option – the provided Dockerfile builds a reproducible environment: docker build -t graphgen . Configuration Copy the example env file and provide API keys for the LLM back‑ends you intend to use (OpenAI, Ollama, etc.). cp .env.example .env # edit .env → set OPENAIAPIKEY, OLLAMAHOST, HFTOKEN, etc. Generation scripts expect a YAML config (e.g. examples/generate/generateatomicqa/atomicconfig.yaml). Adjust model, temperature, and KG source paths as needed. Running a generation The shell wrappers call the Python engine directly. For an atomic QA run: cd examples/generate/generateatomicqa ./generateatomic.sh

The script executes something akin to: python -m graphgen.engine --config atomicconfig.yaml

Output JSON files are written to examples/outputexamples/ (see aggregatedchatml.json, atomicalpaca.json). Evaluation (optional) Use the evaluation helpers, e.g.: cd examples/evaluate/evaluateqa ./evaluateqa.sh # consumes the generated QA and reports metrics

Real‑World Use

A biotech firm can feed internal assay reports (.pdf, .txt) into the KG builder (lightragkgbuilder.py), generate domain‑specific QA (atomic, multi‑hop, VQA) via the appropriate generator, and then fine‑tune a LLaMA‑2‑7B model with LLaMA‑Factory. The pipeline is fully scriptable, making it easy to embed in a nightly data‑augmentation job.

pseudo‑pipeline (YAML for orchestration) steps: name: Build KG run: python -m graphgen.engine --config kgbuild.yaml name: Generate QA run: python -m graphgen.engine --config atomicconfig.yaml name: Fine‑tune run: llamafactory train --model llama2-7b --data generatedqa.json

Code Health & Issues

Medium – No test suite – The repository contains no tests/ directory and CI only runs pylint and packaging checks (.github/workflows/pylint.yml). Uncovered code paths increase regression risk. Medium – External secret exposure risk – .env.example hints at required API keys, but the repo does not enforce secret validation; missing keys will cause runtime failures. Low – Limited type safety – Core data‑flow classes (base* and concrete models) lack explicit type hints; static analysis is limited to linting. Low – Documentation gaps – While README.md and the GitBook cookbook cover high‑level usage, per‑module docstrings are sparse, making onboarding for new contributors slower. Low – Dependency hygiene – The Dockerfile pins major packages, but there is no requirements.txt or pyproject.toml at the repo root, complicating reproducible pip installs outside Docker. Low – License present – LICENSE is included, satisfying legal compliance.

Overall, the code follows a modular design (clear base classes, separate LLM wrappers) and CI enforces linting, but the lack of automated tests and explicit dependency manifest are the most notable weaknesses.

The Bottom Line

GraphGen provides a concrete, extensible pipeline for knowledge‑graph‑driven synthetic QA generation, with ready‑made scripts that can be integrated into fine‑tuning workflows. It is well‑structured but presently lacks automated tests and a clear Python‑package dependency list, so teams should allocate time for validation and possibly add their own test coverage before production use. Ideal for organizations that already manage LLM fine‑tuning and need a systematic way to augment data with domain‑specific knowledge.