credit ocr system - Moses Yebei

The Problem

Credit evaluation teams process 15–20 page loan applications by hand, taking hours per case. This repo automates the document pipeline: OCR text extraction, local LLM field extraction, validation, and stage-based storage — all on local infrastructure with no external AI APIs.

What This Does

src/ocr/label_value_extraction.py extracts key-value pairs from scanned documents using EasyOCR with spatial analysis; src/llm/field_extractor.py validates extracted data against business rules via a local Llama3.1:8b model hosted by Ollama. Results are stored in Azure Blob emulation (Azurite) by processing stage (raw, OCR, LLM, annotated), with metadata in PostgreSQL.

The system is built as a tutorial: nine notebooks under notebooks/ walk from infrastructure setup through a Flask API (src/api/main.py, src/api/routes.py) with Celery async workers (src/tasks/pipeline_tasks.py, src/celery_app.py).

How It Is Wired

Execution starts at run in start_credit_ocr.py:281, which reaches 7 functions. The shortest path from entry to leaving the process: run -> check_prerequisites (subprocess.run) and start -> _setup_database (psycopg2.connect). A full run starts Docker services, waits for them, then serves the API via run_api.py:27 (main).

Everything routes through src/storage/storage.py: get_storage is called from 21 places, upload_blob from 9, blob_path and get_document from 8 each. That module is the dependency hub — 12 modules import it, and churn there is high-blast-radius. The most unstable module is src/integration/pipeline (Ca 5, Ce 9, instability 0.64), which orchestrates OCR → LLM → visualization.

Effects outside the process: 37 functions make outbound network calls, 20 read/write a database, 12 run external commands, 5 touch files. src/dms/adapters.py owns database I/O; src/dms/service.py computes SHA-256 and handles document upload/retrieval; src/api/routes.py wires HTTP to the pipeline.

How To Use It

git clone https://github.com/moses-y/credit-ocr-system
cd credit-ocr-system
# install (pyproject.toml + uv.lock present)
pip install -e .        # or: uv sync
docker compose up -d    # PostgreSQL, Redis, Ollama, Azurite
python start_credit_ocr.py   # starts infrastructure
python run_api.py             # serves the Flask API

Configuration lives in config/credit-ocr-system.conf and config/document_types.conf; copy .env.template to .env for credentials. The README directs setup through notebooks/1-setup/01_setup.ipynb.

Real-World Use

A loan officer uploads data/loan_application.pdf via the API. upload_document stores the blob, Celery runs integrated_pipeline (OCR → LLM extraction → visualization), results land in stage-based containers, and the officer reviews extracted fields with confidence scores and bounding-box overlays in the web UI (src/api/templates/index.html).

Code Health & Issues

Static analysis found 19 findings (2 high, 17 medium). High: deep nesting (max depth 6, in src/dms/adapters.py, start_credit_ocr.py, src/api/routes.py) and 48 duplicated 6-line blocks across 11 files. Notable mediums: broad exception handling in src/storage/storage.py, src/dms/adapters.py, src/dms/service.py; the storage.py hub (12 dependents); high branching in src/llm/validation.py and src/ocr/label_value_extraction.py (23 branch points over 74 lines).

SDLC gaps: no CI workflow, no LICENSE (all rights reserved by default), Dockerfile runs as root with an unpinned base image (python:3.11-slim), and no Dependabot. Note: uv.lock exists despite the "no lockfile" finding — verify which dependency tool is canonical. Notebook 02_ocr_text_extraction.ipynb is 1MB, dominated by stored outputs.

The Bottom Line

A well-structured local-first OCR pipeline that demonstrates a complete credit-document workflow cleanly. It is tutorial-grade, not production-grade: no CI, no license, root containers, and a hot storage hub. Use it to learn the architecture or as a scaffold, not as-is in production.

What the analyser found

Deployment readiness

4/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

90 files

Python46
Markdown12
Jupyter Notebook9
YAML1
SQL1
TOML1

FlaskDocker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["src<br/>35 modules"] g1["tests<br/>9 modules"] g2["(root)<br/>2 modules"] g1 -->|23| g0 g2 -->|1| g0

Module dependencies

The 10 most depended-upon modules of 46, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_src_storage_storage["storage"] n1_src_dms_adapters["adapters"] n2_src_dms_service["service"] n3_src_integration_pipeline["pipeline"] n4_ualization_ocr_visualization["ocr_visualization"] n5_src_config___init__["__init__"] n6_rc_integration_orchestration["orchestration"] n7_src_config_system["system"] n8_src_api_models["models"] n9_src_llm_config["config"] n6_rc_integration_orchestration --> n3_src_integration_pipeline n6_rc_integration_orchestration --> n4_ualization_ocr_visualization n6_rc_integration_orchestration --> n0_src_storage_storage n3_src_integration_pipeline --> n9_src_llm_config n3_src_integration_pipeline --> n0_src_storage_storage n3_src_integration_pipeline --> n2_src_dms_service n3_src_integration_pipeline --> n1_src_dms_adapters n3_src_integration_pipeline --> n7_src_config_system n4_ualization_ocr_visualization --> n0_src_storage_storage

Findings

MediumNo CI/CD pipeline detected - no automated build/test gate .github/ or CI config
MediumNo LICENSE file - unclear usage/redistribution rights root
LowDependencies declared without a lockfile - non-reproducible builds pyproject.toml

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high3

medium5

Add a LICENSE; redistribution rights are undefinedhigh

no licence file at the repository root

Fix: Add MIT or Apache-2.0 at the repository root.

Add a workflow that builds and tests this repositoryhigh

46 source files, no CI configuration

Every change merges with nobody having run the build once, which across this estate is 499 repositories.

Fix: Add a workflow running the project build and test command on push and pull_request.

Add a build gate for the deployable artifacts herehigh

Dockerfile

deployment artifacts with no workflow

Something in the repo is meant to reach a server and no automated step has ever validated it, which is the state of 125 repositories here.

Fix: Add a workflow that at least builds the image and validates the manifests.

Enable Dependabot or Renovatemedium

1 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

python:3.11-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Strip notebook outputs before committingmedium

notebooks/2-ocr-based-text-extraction/02_ocr_text_extraction.ipynb

1 notebook(s) over 1MB, 1.0MB total, dominated by stored output

Outputs are where a printed dataframe of customer rows or a traceback containing a connection string ends up, and diff review never reads them.

Fix: Install nbstripout and clear the existing outputs once.

Split the oversized notebook cell into stepsmedium

notebooks/2-ocr-based-text-extraction/02_ocr_text_extraction.ipynb

largest cell 8691 characters of 38510 (3 notebooks sampled)

A failure anywhere in a 22,000-character cell means rerunning the whole pipeline from load to model, so nothing can be debugged in isolation.

Fix: Break at the load, clean, feature, model and evaluate boundaries.

Add a non-root USER to the imagemedium

Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.