May 19, 2025 5 min read Python original

retail analytics

Abstract

This repo contains Retail Analytics with LLMs for sentiment analysis

The Problem

Retail teams need a single place to turn raw sales, customer, and review data into forecasts, segmentations, and sentiment insights. Building and maintaining separate pipelines, APIs, and dashboards for each analytic task is time‑consuming and error‑prone.

What This Does

The repository is a portfolio of four self‑contained projects:

Project	Core code	Main purpose
`src/`	data preprocessing, model training, utilities, visualisation (`src/data/`, `src/models/`, `src/utils/`, `src/visualization/`)	End‑to‑end pipelines for sales forecasting, customer segmentation, and review‑sentiment modelling.
`api/`	FastAPI entry point (`api/main.py`) and routers (`api/routers/*.py`)	HTTP layer exposing model inference, data retrieval, and RAG‑based Q&A.
`dashboard/`	Streamlit app (`dashboard/app.py`) and UI components (`dashboard/components/.py`, `dashboard/pages/.py`)	Interactive visualisation of forecasts, segmentations, and sentiment dashboards.
`notebooks/`	Exploratory notebooks (`notebooks/*.ipynb`)	Prototype analyses and visual output for data‑science validation.

Key files: src/cli.py (CLI that drives training/evaluation), api/models.py (validation helpers), src/models/sentiment.py (LLM‑based sentiment pipeline), src/models/forecasting.py (XGBoost/LightGBM/Prophet), src/models/segmentation.py (KMeans/DBSCAN), dashboard/components/charts.py (plot utilities), src/utils/config.py (central config loader).

How It Is Wired

CLI entry points (src/cli.py) – train_models, evaluate_models, preprocess_data. train_models → loads config (load_config in src/utils/config.py – 11 callers) → prepares data (prepare_sentiment_data, prepare_forecasting_data) → calls model‑specific trainers (train_sentiment_model, train_xgboost_model, train_kmeans_model). evaluate_models → loads test data (load_data) → runs evaluate_models → persists metrics (save_plot – called from 14 places). * preprocess_data → reads raw CSV/JSON (load_data), cleans (clean_retail_sales_data), writes processed files (save_processed_data).

API server (api/main.py) starts a FastAPI app. Routers (api/routers/*.py) import helpers from api/dependencies.py (13 functions, 2 classes). Typical request flow: /forecast → api/routers/forecasting.py → api/dependencies.get_forecasting_model → src/models/forecasting.get_model_config → model inference (model.predict). /sentiment → api/routers/reviews.py → src/models/sentiment.train_sentiment_model (or inference) → returns sentiment scores.

Dashboard (dashboard/app.py) launches Streamlit. UI callbacks call the same utility functions used by the API (load_config, save_plot, generate_embeddings). The only external call is a requests call without a timeout (identified as a medium issue).

Hot‑spot modules (most inbound imports):

dashboard/components/charts.py (Ca = 5) – central plot creator.
src/utils/config.py (Ca = 6) – sole source of configuration loading.

Call‑graph highlights – save_plot is invoked from 14 distinct locations, making it a high‑impact change point. train_sentiment_model and train_kmeans_model each have 8–7 callers, indicating tightly coupled training logic.

No circular imports were detected; the import graph is shallow (47 edges across 57 modules). The most unstable module by metric is src/cli.py (instability = 1) because it imports many others but is not imported itself.

How To Use It

# Clone the repo (exact URL as requested)
git clone https://github.com/moses-y/retail-analytics.git
cd retail-analytics

# Virtual‑env install (as per README)
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
uv pip install -r requirements.txt   # uv is used in the README

# Copy and edit environment variables
cp .env.example .env
# Edit .env to set DB paths, API keys, etc.

# Run the API
uvicorn api.main:app --reload --port 8000

# Run the dashboard
streamlit run dashboard/app.py

# Or build/run everything with Docker
docker compose up -d

Configuration lives in config/ (api_config.yml, model_config.yml, monitoring_config.yml). The CLI and API both call src.utils.config.load_yaml_config to read these files. No secret values are committed; the .env.example shows required keys.

Real‑World Use

A retailer schedules a nightly CI job that executes python src/cli.py train_models. The script reads config/model_config.yml, trains the XGBoost forecasting model and a KMeans segmentation model, and writes serialized artifacts to models/. The FastAPI service then serves /forecast and /segment endpoints for downstream inventory‑management systems, while the Streamlit dashboard visualises the latest forecasts for business users.

Code Health & Issues

High – Pin GitHub Actions to commit SHAs (.github/workflows/*).
Medium – Declare least‑privilege permissions for GITHUB_TOKEN (.github/workflows/cd.yml).
Medium – Enable Dependabot/Renovate for dependency updates.
Medium – Pin Docker base image by digest (Dockerfile).
Medium – Add a dependency‑vulnerability scan step to CI.
Medium – Split oversized notebook cell (notebooks/sales_forecasting_customer_segmentation.ipynb).
Medium – Add timeout to outbound requests call (dashboard/app.py).
Medium – Set persist-credentials: false on checkout (.github/workflows/cd.yml).
Medium – Run container as non‑root user (Dockerfile).
Medium – Pin vulnerable package resolutions by committing a lockfile (currently none).
Low – Deep nesting (max indentation depth = 8) in api/models.py, src/data/preprocessing.py, src/utils/metrics.py.
Medium – Broad exception handling in api/dependencies.py and src/models/forecasting.py.
High – Duplicated 6‑line blocks across routers and dashboard pages.
Medium – Oversized files (src/visualization/plots.py, api/routers/reviews.py).

The repo includes tests (tests/), CI (.github/workflows/), a Dockerfile, and an MIT licence, but lacks a lockfile and several hardening practices noted above.

The Bottom Line

The codebase delivers a functional end‑to‑end retail analytics stack with clear separation between pipelines, API, and UI. It is usable out‑of‑the‑box but suffers from maintainability concerns (deep nesting, duplicated blocks) and security hygiene gaps (unpinned actions, no lockfile). Engineers comfortable refactoring Python code and tightening CI/CD can adopt it quickly; teams that need a production‑grade, reproducible build should first address the high‑severity health items.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

118 files

Python57
Markdown11
YAML8
Jupyter Notebook3

FlaskDocker

Architecture

Top-level areas of the codebase, sized by module count. Arrows show how many imports cross from one area into another.

graph TD g0["src<br/>26 modules"] g1["tests<br/>11 modules"] g2["api<br/>10 modules"] g3["dashboard<br/>9 modules"] g3 -->|10| g0 g1 -->|9| g0 g1 -->|3| g2 g0 -->|1| g2

Module dependencies

The 10 most depended-upon modules of 57, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_src_utils_config["config"] n1_dashboard_components_charts["charts"] n2_src_data_preprocessing["preprocessing"] n3_api_dependencies["dependencies"] n4_dashboard_components_filters["filters"] n5_api_models["models"] n6_src_models_forecasting["forecasting"] n7_src_models_segmentation["segmentation"] n8_src_models_sentiment["sentiment"] n9_api_main["main"] n9_api_main --> n3_api_dependencies

Dependencies

15 of 32 declared dependencies are a major version behind. Resolved 32 of 32 against the registry.

Package	Declared	Latest	Behind
`sentence-transformers`	>=2.2.2	5.7.0	3 major
`black`	>=23.3.0	26.5.1	3 major
`isort`	>=5.12.0	8.0.1	3 major
`pandas`	>=1.5.3	3.0.5	2 major
`xgboost`	>=1.7.5	3.4.1	2 major
`pytest`	>=7.3.1	9.1.1	2 major
`numpy`	>=1.24.3	2.5.2	1 major
`plotly`	>=5.14.1	6.9.0	1 major

Findings

LowDependencies declared without a lockfile - non-reproducible builds requirements.txt

Open in Code Brain → See its neighbourhood →

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high1

medium9

low1

Pin third-party GitHub Actions to a commit SHAhigh8 occurrences

.github/workflows

codecov/codecov-action@v3, docker/setup-buildx-action@v2, docker/login-action@v2, docker/metadata-action@v4

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Declare least-privilege permissions for GITHUB_TOKENmedium3 occurrences

.github/workflows/cd.yml

3 workflow(s) declare no permissions, 2 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Enable Dependabot or Renovatemedium

1 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

python:3.10-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Split the oversized notebook cell into stepsmedium

notebooks/sales_forecasting_customer_segmentation.ipynb

largest cell 22622 characters of 22622 (3 notebooks sampled)

A failure anywhere in a 22,000-character cell means rerunning the whole pipeline from load to model, so nothing can be debugged in isolation.

Fix: Break at the load, clean, feature, model and evaluate boundaries.

Give the outbound request a timeoutmedium

dashboard/app.py

1 outbound call(s) with no timeout

requests has no default timeout, so a peer that accepts the connection and then says nothing holds the worker forever, and with a small worker pool a single slow dependency takes the whole service down without anything appearing to fail.

Fix: Pass timeout= on every call, or use a session with one configured.

Set persist-credentials: false on checkoutmedium

.github/workflows/cd.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Pin the resolution so a vulnerable version cannot installmedium231 occurrences

no lockfile, and the declared ranges reach lightgbm@3.3.5 CVE-2024-43598 (critical) +1 more; transformers@4.28.1 CVE-2023-6730 (critical) +49 more; nltk@3.8.1 CVE-2025-14009 (critical) +30 more; and 4 more package(s)

The range includes versions with published advisories and nothing in the repository records which one was resolved, so two installs a month apart can differ in whether they are vulnerable and neither is reproducible from what is committed.

Fix: Commit the lockfile, which fixes the resolution and makes this answerable.

Set timeout-minutes on the workflow jobslow3 occurrences

.github/workflows/cd.yml

3 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.

View on GitHub All Projects