The Problem Fine‑tuning large language models (LLMs) with high‑quality, domain‑specific QA data is expensive and brittle. Practitioners need a reproducible way to generate synthetic QA pairs that target knowledge gaps identified in a model’s calibration, while keeping the pipeline modular and inspectable.

What This Does graphgen/ implements the full synthetic‑data workflow:

  • Knowledge graph construction – graphgen/models/kg_builder/*.py builds fine‑grained graphs from source texts.
  • Gap detection – graphgen/utils/detect_lang.py (e.g., detect_main_language) and the BaseSearcher hierarchy (graphgen/bases/base_searcher.py) compute expected calibration error and surface under‑represented facts.
  • Targeted generation – LLM wrappers in graphgen/models/llm/local/*.py (e.g., hf_wrapper.py, vllm_wrapper.py) are called by the generator classes (graphgen/models/generator/*.py) such as aggregated_generator.py and cot_generator.py.
  • Post‑processing & storage – graphgen/common/init_storage.py creates a KV store (NetworkX or RocksDB) and graphgen/bases/base_storage.py provides a thin API used throughout the pipeline.

The baseline scripts under baselines/ (EntiGraph, Genie, etc.) show how to invoke the same generators with minimal wiring.

How It Is Wired

  1. Entry point – graphgen/run.py defines main() (line 32) which parses CLI arguments and calls the high‑level execute function in graphgen/engine.py.
  2. engine.execute (line 268) orchestrates the workflow: it initializes the LLM (graphgen/common/init_llm.py → init_llm), the storage backend (graphgen/common/init_storage.py → init_storage), and then iterates over configured generators.
  3. Core functions with the widest blast radius: graphgen/utils/__init__.py – imported by 57 other modules (instability 0.14). graphgen/bases/__init__.py – imported by 54 modules (instability 0.22). * graphgen/bases/datatypes.py – imported by 27 modules (instability 0).
  4. Frequently called internal APIs (most distinct callers): generate_answer – 6 × from generate → generate_answer. detect_main_language – 11 × from build_prompt. * partition – 10 × across partitioner implementations.
  5. External effects: Filesystem writes (os.makedirs) in set_working_dir (called from main). Network calls in LLM wrappers (graphgen/models/llm/api/openai_client.py) and searchers (graphgen/models/searcher/db/ncbi_searcher.py). * Model inference in graphgen/bases/base_llm_wrapper.py.

No circular import cycles were detected, but the hub modules above are high‑impact change points; altering them will cascade through dozens of files.

How To Use It

# Clone (use the exact URL as required)
git clone https://github.com/moses-y/GraphGen
cd GraphGen

# Install dependencies (editable mode preserves local edits)
pip install -e .   # reads requirements.txt / pyproject.toml

# Prepare environment – copy the template and fill in keys
cp .env.example .env
# Edit .env: set SYNTHESIZER_API_KEY, any storage paths, etc.

# Run the CLI (example: generate aggregated QA)
python -m graphgen.run --generator aggregated --config examples/generate/generate_aggregated_qa/aggregated_config.yaml

For containerised deployment:

docker build -t graphgen:latest .
docker run --env-file .env -v $(pwd)/data:/data graphgen:latest

The web UI can be started with python webui/app.py (listens on Flask’s default port) after the same pip install -e . step.

Real‑World Use A data‑science team can point graphgen/run.py at a corpus of domain papers, let the KG builder extract entities, and automatically generate multi‑hop QA pairs. The resulting JSONL files feed directly into LLaMA‑Factory or xtuner for SFT, reducing manual annotation effort by >50 % as reported in the paper.

Code Health & Issues

Measured findings (high‑impact):

  • Hub modules – graphgen/utils/__init__.py, graphgen/bases/__init__.py, graphgen/bases/datatypes.py are imported by >50 files; keep them stable.
  • Deep nesting – 25 files (e.g., graphgen/bases/base_splitter.py) have indentation depth ≥ 7; refactor into early‑return helpers.
  • Duplicated blocks – repeated 6‑line snippets across 44 baseline files; extract shared helpers.
  • Broad exception handling – generic except: in several searcher and utility modules; replace with specific catches.
  • High branching density – graphgen/storage/kv/json_storage.py and similar have >15 branches in <60 lines; consider strategy pattern.
  • Oversized file – webui/app.py (604 LOC); split by responsibility (routes, utils, config).

Code‑health audit (from static analysis):

  • HIGH Pin GitHub Actions to commit SHA – .github/workflows/*.yml.
  • HIGH Add test execution to CI – existing workflows lack a test step.
  • HIGH Use PR workflow instead of direct push to default – .github/workflows/push-to-hf.yml.
  • HIGH Remove secret from Docker build args – Dockerfile embeds SYNTHESIZER_API_KEY.
  • MEDIUM Declare least‑privilege GITHUB_TOKEN permissions – same workflow file.
  • MEDIUM Enable Dependabot/Renovate – no manifest watcher configured.
  • MEDIUM Pin base Docker image by digest – python:3.10-slim.
  • MEDIUM Add dependency‑vulnerability gate – no scan step present.
  • MEDIUM Set persist-credentials: false on checkout – to avoid token leakage.
  • LOW Set explicit timeout-minutes on workflow jobs – currently missing.

The Bottom Line GraphGen provides a reasonably complete, modular pipeline for knowledge‑driven synthetic QA generation, with clear entry points and extensible LLM wrappers. However, the codebase contains several high‑impact hubs, deep nesting, and duplicated logic that increase maintenance risk, and the CI/CD configuration needs tightening for security and reliability. It is suitable for research teams comfortable with Python, Docker, and LLM APIs, but production users should address the highlighted health issues first.