The Problem

Data‑science teams need a reproducible way to define, materialize, and serve feature pipelines at scale while keeping the same logic across batch, streaming, and online inference. Maintaining separate codebases for transformation, registry, and UI quickly becomes a coordination nightmare and introduces hidden drift between development and production.

What This Does

Feathr is a modular platform that bundles the core feature‑store engine (feathr-impl), configuration helpers (feathr-config), a Python‑centric client library (feathr_project), a React UI (ui), and three registry back‑ends (registry/*). The Python client (feathr_project/feathrcli/cli.py) lets users declare feature transformations in pure Python, registers them via the registry services (registry/purview-registry/main.py, registry/sql-registry/main.py), and materializes results on Spark clusters (feathr-compute). The UI (ui/src/components/...) provides a visual catalog and lineage view. All components are packaged as Docker images (FeathrSandbox.Dockerfile, FeathrRegistry.Dockerfile) and orchestrated through Gradle builds (feathr-impl/build.gradle, etc.).

How It Is Wired

  • Entry point – feathr_project/feathrcli/cli.py parses the CLI, loads feathr_project/feathr/client.py, and builds a FeatureRegistry object.
  • Client → Registry – client.py imports feathr_project/feathr/registry/_feature_registry_purview.py (the largest file, 1,222 LOC) or the SQL variant, which contact the respective service via HTTP/REST.
  • Registry → Compute – Registry services (registry/*/main.py) forward materialization requests to the compute layer (feathr-compute/src/main/scala/...) which launches Spark jobs using feathr_project/feathr/spark_provider/_databricks_submission.py.
  • Compute → Data – Spark jobs read raw sources (e.g., Hive, Azure Blob) and write feature tables back to the same storage, leveraging the pandas‑based utilities in feathr_project/feathr/utils/.
  • UI ↔ Registry – The React UI (ui/src/components/FlowGraph/...) calls the registry REST endpoints to list features and display lineage. The most‑connected UI module is ui/src/components/FlowGraph/interface (4 inbound imports, 0 outbound).
  • Dependency graph – 222 internal modules, 66 import edges, no cycles. The hub feathr_project/feathr/__init__ imports 20 other modules but is not imported itself, indicating a low‑risk, low‑coupling entry point.
  • Build & Deploy – Gradle compiles the Scala/Java core (feathr-impl, feathr-config, feathr-data-models). The UI is built with npm (package.json implied by the presence of npm as the package manager). Dockerfiles produce self‑contained images; the sandbox image is the recommended quick‑start.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/feathr.git
cd feathr

# Build core libraries (requires Java 11+ and Gradle)
./gradlew build            # compiles all *.gradle projects

# Build the UI (requires Node 16+)
cd ui && npm ci && npm run build && cd ..

# Build the sandbox container (includes UI, client, compute)
docker build -f FeathrSandbox.Dockerfile -t feathr-sandbox:latest .

# Run the sandbox (ports: UI 80, Jupyter 8888, Interpret 7080)
docker run -it --rm -p 8888:8888 -p 8081:80 -p 7080:7080 \
  -e GRANT_SUDO=yes feathr-sandbox:latest

Configuration – The sandbox expects environment files under registry/access_control/.env and ui/.env. Those files are currently committed and must be removed (see health section). Production deployments replace them with secure secret stores (Azure Key Vault, etc.).

Running a feature pipeline – After the container is up, use the CLI inside the sandbox:

python -m feathrcli register -f examples/feature_definition.py
python -m feathrcli materialize -f my_feature_anchor

The CLI invokes feathr_project/feathr/client.py, which talks to the registry service and triggers a Spark job via the compute module.

Real‑World Use

A data‑science team can define a time‑window aggregation in my_feature_anchor.py, register it with feathrcli register, and then schedule nightly materializations through Azure Data Factory. The UI shows the feature lineage, allowing model owners to audit point‑in‑time joins before training. Because the same Python definition drives both offline materialization and online serving, drift is eliminated.

Code Health & Issues

Measured static‑analysis findings (97 total)

  • High – Deep nesting (46 instances, e.g., feathr_project/feathr/definition/feature.py) – flatten with early returns.
  • High – Duplicated blocks (1570 lines across 217 files) – extract shared helpers.
  • High – Oversized files (3 files > 1 k LOC) – split into focused modules.
  • Medium – Broad exception handling (8 files) – catch specific exceptions.
  • Medium – File opened without context manager (2 files) – use with open(...).

Code‑health audit (12 findings)

  • High – Pin GitHub Actions to commit SHAs (.github/workflows/*).
  • High – Add a lockfile beside feathr_project/pyproject.toml.
  • High – Remove committed .env files (registry/access_control/.env, ui/.env*) and rotate credentials.
  • High – CI does not run the test suite; add a test step to the existing workflows.
  • Medium – Declare least‑privilege GITHUB_TOKEN permissions.
  • Medium – Enable Dependabot or Renovate for dependency updates.
  • Medium – Pin Docker base images by digest (registry/purview-registry/Dockerfile).
  • Medium – Add a dependency‑vulnerability scan to PR checks.
  • Medium – Expand the README (currently 236 bytes).
  • Medium – Set persist-credentials: false on the checkout step of the PyPI publish workflow.

No critical findings were reported, but the presence of committed secrets and missing CI test execution are immediate blockers for production use.

The Bottom Line

Feathr delivers a complete, enterprise‑grade feature‑store stack with clear separation of concerns across compute, registry, client, and UI. The codebase is functional but suffers from maintainability issues (deep nesting, duplicated logic) and several security‑related hygiene problems that must be remedied before adopting it in a regulated environment. Teams comfortable managing multi‑language builds (Java/Scala, Python, TypeScript) and Docker deployments can extract immediate value, provided they address the high‑severity health findings.