Clone via: git clone https://github.com/moses-y/langextract

LangExtract (moses‑y fork)

Overview

LangExtract is a Python library that uses LLMs to extract structured information from unstructured text, with precise source grounding and interactive HTML visualisation. It is a fork of google/langextract (original 38 k stars) and currently has 0 stars. The code base comprises 129 files (46 in langextract, 25 tests, 19 GitHub‑Actions workflows, etc.). Primary language Python (79 files). The library supports cloud models (Gemini, OpenAI) and local Ollama models, and can produce interactive HTML visualisations of extractions.

Quick start

Define a prompt and a high‑quality example, then call langextract.extract. Example from the README:

import langextract as lx
import textwrap

prompt = textwrap.dedent("""\
    Extract characters, emotions, and relationships …
""")
# provide example data...
documents = [...]
lx.extract(prompt, documents)   # entry point from `langextract/__init__.py`

The function extract lives in langextract/__init__.py and routes through the internal call graph to tokenisation, inference and resolver steps.

Measured code‑health findings (static analysis, 31 items)

  • Import cycle – langextract/__init__.py and langextract/extraction.py form a circular dependency (instability 0). Break by extracting shared types or deferring imports.
  • Hub modules – langextract/__init__.py, langextract/core/__init__.py, langextract/providers/__init__.py are imported by 35‑45 other modules; churn here has high blast‑radius. Keep them small; move volatile logic out.
  • Duplicated CI scripts – 58 repeated 6‑line blocks across .github/scripts/add-new-checks.sh, add-size-labels.sh, revalidate-all-prs.sh, zenodo_publish.py, etc. Extract a shared helper.
  • Deep nesting – Max indentation depth 8 in examples/custom_provider_plugin/langextract_provider_example/schema.py, tests/schema_test.py, scripts/create_provider_plugin.py. Flatten with guard clauses.
  • Oversized files – tests/annotation_test.py (1237 lines), tests/resolver_test.py, langextract/providers/gemini_batch.py. Split by responsibility.
  • Broad exception handling – Bare except in langextract/core/debug_utils.py, langextract/providers/gemini_batch.py, langextract/visualization.py. Catch specific exceptions.
  • High branching density – scripts/validate_community_providers.py has 44 branch points over 145 lines. Decompose decision logic.

Code health audit (12 findings, ranked)

  • [CRITICAL] PR checkout in privileged workflow – .github/workflows/ci.yaml checks out the untrusted PR ref, enabling credential exfiltration. Split into an unprivileged pull_request build and a workflow_run that only consumes the PR artifact.
  • [CRITICAL] Secrets in fork‑triggered workflows – GEMINI_API_KEY, OPENAI_API_KEY reachable from contributor‑controlled steps. Move secret‑using steps into a workflow_run job that never checks out PR code, or gate on required reviewers.
  • [HIGH] Pin third‑party GitHub Actions to commit SHA – tj-actions/changed-files@v46, pypa/gh-action-pypi-publish@release/v1 use version tags that can move. Replace with 40‑character commit SHAs.
  • [HIGH] Commit a lockfile beside the manifest – pyproject.toml has no lockfile; unlocked ranges may ship different transitive code. Run the package manager once and commit the generated lockfile.
  • [HIGH] Remove continue-on-error from correctness gating – CI line 78 a test step uses continue-on-error; a failing suite would report green. Delete it or move the step to a non‑required job.
  • [MEDIUM] Enable Dependabot / Renovate – No update bot configured for Python or GitHub‑Actions ecosystems. Add .github/dependabot.yml.
  • [MEDIUM] Pin container base image by digest – Dockerfile uses python:3.10-slim (mutable tag). Prefer python:3.10-slim@sha256:<digest> and enable Docker Dependabot.
  • [MEDIUM] Gate pull requests on dependency vulnerability scan – No dependency‑scan step in CI. Add dependency-review-action on pull_request or osv-scanner on push.
  • [MEDIUM] Move large binaries to Git LFS – docs/_static/romeo_juliet_full.gif is 9.0 MB. Track *.gif (or larger) with Git LFS, or fetch in a setup step.
  • [MEDIUM] Set persist-credentials: false on checkout – .github/workflows/publish.yml checkout retains the token. Add with: persist-credentials: false and pass explicit tokens only to the push step.
  • [LOW] (audit) – (not listed further)

The bottom line

LangExtract offers a pragmatic path to LLM‑driven extraction with source‑level grounding and visual replay, valuable for teams that need traceable structured data from clinical notes, reports, or long documents. The code base shows several high‑impact maintainability concerns—circular imports, outsized files, duplicated CI logic—that should be addressed before scaling. Security‑wise, the CI workflows expose secrets and allow PR‑code checkout in privileged contexts; those need immediate remediation. Teams comfortable with adding a lockfile, pinning Actions SHAs, and restructuring a few hot‑path modules will get the most stable experience. If you need a ready‑to‑use extraction pipeline today, the library works, but plan to invest in the listed hygiene fixes for long‑term reliability.