The Problem

Deploying a hand‑gesture classifier that works reliably across skin tones and lighting conditions usually requires a large CNN and GPU resources. Teams that need a lightweight, CPU‑only solution for real‑time ASL alphabet detection lack a ready‑made pipeline.

What This Does

The repository delivers a proof‑of‑concept that extracts 21 MediaPipe hand landmarks (63 × float features), normalises them to the wrist, and classifies the resulting vector with a pre‑trained XGBoost model (asl_model_xgb.pkl).

  • Retrained_with_XGBOOST.ipynb defines the core preprocessing routine normalize_landmarks.
  • sign_detection.ipynb loads asl_model_xgb.pkl and label_encoder.pkl, opens a webcam with OpenCV, runs MediaPipe’s HandLandmarker, applies normalize_landmarks, and calls the XGBoost model for inference.
  • The notebook visualises predictions on‑screen and includes a static confusion matrix (confusion_matrix.png) for reference.

How It Is Wired

  1. Entry point – execution starts in sign_detection.ipynb. The notebook imports the helper function normalize_landmarks (originating in Retrained_with_XGBOOST.ipynb).
  2. External touchpoints – MediaPipe’s hand‑landmark solution performs a one‑time download of its binary model (outbound network call). The XGBoost classifier (predict) is the sole inference call to a local model file.
  3. Data flow cv2.VideoCapture → frame image. MediaPipe HandLandmarker.process(frame) → raw landmarks. normalize_landmarks(raw_landmarks) → wrist‑centred, scale‑invariant vector (63 floats). xgb_model.predict(normalized_vector) → class index → label_encoder.inverse_transform → printable letter.
  4. Effect ownership – all side‑effects (camera I/O, model loading, network download) are confined to sign_detection.ipynb. No other modules import or reuse the function, so the blast radius is limited to this notebook. Because the code lives in notebooks, there is no explicit package graph; any refactor must first extract normalize_landmarks and the inference loop into reusable modules.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/asl-sign-detection-mediapipe-rf
cd asl-sign-detection-mediapipe-rf

# Install Python dependencies
python -m pip install -r requirements.txt

# Launch the demo notebook
jupyter notebook sign_detection.ipynb

Inside the notebook run the cells sequentially. No additional configuration files, environment variables, or API keys are required. The webcam will open, and the predicted ASL letter will be overlaid on each frame.

Real‑World Use

A tele‑health platform could embed the inference loop in a lightweight Flask route:

from flask import Response, stream_with_context
import cv2, mediapipe as mp
from xgboost import XGBClassifier
from sklearn.preprocessing import LabelEncoder

model = XGBClassifier()
model.load_model('asl_model_xgb.pkl')
le = LabelEncoder()
le.classes_ = joblib.load('label_encoder.pkl')

def gen_frames():
    cap = cv2.VideoCapture(0)
    while True:
        ret, frame = cap.read()
        lm = mp.solutions.hands.HandLandmarker().process(frame).landmarks
        norm = normalize_landmarks(lm)
        pred = le.inverse_transform(model.predict([norm]))[0]
        # draw pred on frame …
        yield (b'--frame\r\n'
               b'Content-Type: image/jpeg\r\n\r\n' + jpeg_bytes + b'\r\n')

The route streams annotated video to a browser, providing on‑device ASL transcription without a GPU.

Code Health & Issues

  • Medium – Enable Dependabot – one manifest (requirements.txt) present, no automated update bot configured.
  • No test suite (tests folder missing).
  • No CI/CD pipeline (.github/ workflows absent).
  • No LICENSE file – redistribution rights unclear.
  • No lockfile (requirements.txt only) – builds are not reproducible.

All items stem from the static analysis results; no additional concerns were detected.

The Bottom Line

The repo showcases a high‑accuracy, CPU‑friendly ASL alphabet recogniser that runs from a notebook with minimal setup. It is suitable for prototyping or educational demos but lacks production‑grade safeguards (tests, CI, licensing, reproducible builds). Teams that need a maintainable, deployable service should extract the notebook logic into a package, add tests, and address the listed SDLC gaps.