The Problem
Clients that need a low‑latency, skin‑tone‑agnostic ASL alphabet recognizer often resort to CNNs that demand GPU resources and large image datasets. Deploying such models on edge devices or modest laptops can be costly and fragile, especially when lighting or background varies.
What This Does
The repository implements a real‑time ASL alphabet detector that bypasses raw‑pixel CNNs. It extracts 21 hand landmarks per frame with MediaPipe HandLandmarker, normalises them relative to the wrist, and classifies the 63‑dimensional feature vector with a pre‑trained XGBoost model (aslmodelxgb.pkl).
Key artefacts:
| File | Role |
|---|---|
| signdetection.ipynb | End‑to‑end pipeline: data loading, landmark extraction, model inference on webcam |
| RetrainedwithXGBOOST.ipynb | Model training script that produces aslmodelxgb.pkl and labelencoder.pkl |
| aslmodelxgb.pkl | Serialized XGBoost classifier (binary pickle) |
| labelencoder.pkl | Scikit‑learn LabelEncoder mapping class indices to letters |
| confusionmatrix.png | Visual validation of the 98.43 % test accuracy reported in the README |
The approach is computationally cheap (hand landmark extraction ≈ 15 ms on CPU) and remains accurate across diverse skin tones because it operates on geometry rather than colour.
How To Use It
Setup – install the exact Python stack listed in requirements.txt. python -m venv .venv source .venv/bin/activate # Windows: .venv\Scripts\activate pip install -r requirements.txt Verify assets – ensure the model files (aslmodelxgb.pkl, labelencoder.pkl) and the optional landmark CSV (hosted externally) are present in the repository root. Run inference – open signdetection.ipynb and execute the cells. The notebook loads the pickled model, starts the default webcam via OpenCV, extracts MediaPipe landmarks for each frame, normalises them, and prints the predicted letter in real time.
If a command‑line entry point is preferred, the notebook’s core logic can be extracted to a script (e.g., app.py), but such a script is not provided in the current repo. (Optional) Retrain – execute RetrainedwithXGBOOST.ipynb. It expects a landmarksdataset.csv (82 MB) referenced in the README; the file must be downloaded from the linked Google Drive location before running.
Real‑World Use
A small‑scale assistive kiosk could embed this pipeline on a Raspberry Pi 4. The kiosk would launch the notebook (or a derived script) at boot, capture a user’s hand via a USB camera, and display the predicted letter on an attached screen. Because the model relies only on CPU‑friendly operations, no GPU is required, keeping hardware costs low.
Minimal wrapper for kiosk deployment
import cv2, mediapipe as mp, joblib, numpy as np from xgboost import XGBClassifier
model = joblib.load('aslmodelxgb.pkl') le = joblib.load('labelencoder.pkl') hands = mp.solutions.hands.Hands(staticimagemode=False, maxnumhands=1, mindetectionconfidence=0.6)
cap = cv2.VideoCapture(0) while cap.isOpened(): , frame = cap.read() lm = hands.process(cv2.cvtColor(frame, cv2.COLORBGR2RGB)).multihandlandmarks if lm: pts = np.array([[p.x, p.y, p.z] for p in lm[0].landmark]).flatten() # wrist‑centred normalisation (same as training) wrist = pts[:3] norm = (pts - np.tile(wrist, 21)) / np.linalg.norm(wrist) pred = le.inversetransform([model.predict([norm])[0]])[0] cv2.putText(frame, pred, (30, 30), cv2.FONTHERSHEYSIMPLEX, 1, (0,255,0), 2) cv2.imshow('ASL', frame) if cv2.waitKey(1) & 0xFF == 27: break cap.release() cv2.destroyAllWindows()
Code Health & Issues
Medium – Missing test suite – No tests/ directory or pytest configuration; model inference paths are untested. Medium – No CI/CD – Repository lacks .github/workflows or other pipeline definitions, so regressions are not automatically caught. Medium – License absent – No LICENSE file despite the badge in README; legal reuse is ambiguous. Low – No lockfile – requirements.txt pins versions loosely; reproducible builds could be affected by upstream updates. Low – Inconsistent documentation – README references app.py and aslmodelxgb.json, which are not present; actual entry point is signdetection.ipynb. This may confuse new users. Low – Large data externalised – landmarks_dataset.csv is required for retraining but not version‑controlled; availability depends on external Google Drive link.
The Bottom Line
The repo delivers a lightweight, high‑accuracy ASL alphabet recognizer that runs on CPU and avoids image‑level deep learning. It is well‑suited for prototypes, edge deployments, or educational demos where GPU resources are scarce. However, the lack of tests, CI, and a clear license limits confidence for production use; a modest amount of engineering effort (add a CLI wrapper, test harness, and proper licensing) would make it a reliable component for commercial assistive‑technology projects.