The Problem

The “For You” feed must merge two very different content sources—posts from accounts a user follows (Thunder) and globally retrieved posts (Phoenix)—and rank them in real‑time. Maintaining low latency while applying a heavy transformer model and many hydration/filter steps creates a high‑maintenance codebase that is hard to reason about and test.

What This Does

The repository is a portfolio of four self‑contained services:

  • home‑mixer – orchestration layer that hydrates queries, runs candidate sources, applies filters and scorers, then returns a ranked list. Core files: home-mixer/main.rs, home-mixer/lib.rs, home-mixer/candidate_hydrators/*, home-mixer/filters/*, home-mixer/scorers/*.
  • thunder – provides in‑network posts and low‑level storage. Entry points: thunder/main.rs and thunder/lib.rs; key logic in thunder/posts/post_store.rs and thunder/kafka/tweet_events_listener_v2.rs.
  • candidate‑pipeline – a reusable pipeline that can be invoked by either home‑mixer or external services. Entry point: candidate-pipeline/candidate_pipeline.rs (function execute). Handles request IDs, source selection, hydration, filtering, scoring and post‑selection.
  • phoenix – Python wrapper around the Grok‑based transformer model used for scoring. Declared in phoenix/pyproject.toml; runtime code lives in phoenix/grok.py and related modules.

All four share a common data model (candidate structs, filters, scorers) defined across the Rust crates.

How It Is Wired

  1. Start – home-mixer/main.rs (line 30) launches the service and calls the orchestration function chain (≈19 functions).
  2. Query Hydration – home-mixer/query_hydrators/user_action_seq_query_hydrator.rs builds a PureCoreData struct (via home-mixer/candidate_pipeline/candidate_features.rs).
  3. Candidate Sources – In‑network: thunder/posts/post_store.rs (new, insert_posts, mark_as_deleted). Out‑of‑network: phoenix/run_retrieval.rs (calls the Grok model in phoenix/grok.py).
  4. Pipeline Execution – candidate-pipeline/candidate_pipeline.rs’s execute function (line 53) is the central hub; it reaches 34 distinct functions, invoking: hydrate → fetch_vf_results (candidate‑pipeline/hydrator.rs) filter → filter (candidate‑pipeline/filter.rs) – called from 10 places overall score → apply (home‑mixer/scorers/phoenix_scorer.rs) – 19 calls from compute_weighted_score select → selector (candidate‑pipeline/selector.rs) * post_selection_hydrators and post_selection_filters (home‑mixer) for final trimming.
  5. Scoring – home-mixer/scorers/phoenix_scorer.rs invokes the Python model via a FFI bridge (not shown in the static graph) and produces engagement probabilities.
  6. Response – home-mixer/server.rs serializes the ranked list back to the client.

Blast‑radius hotspots – filter, request_id, and hydrate appear in the highest fan‑in lists (10, 5, 2 callers respectively). Changing their signatures or side‑effects will ripple through many modules. The import graph shows no circular dependencies, but the call graph contains deep nesting (max depth 8) in files such as thunder/kafka/tweet_events_listener_v2.rs and candidate-pipeline/candidate_pipeline.rs, increasing cognitive load for modifications.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/x-algorithm
cd x-algorithm

# Build all Rust crates
cargo build --workspace

# Run unit tests (only 2 currently)
cargo test --workspace

# Install the Python scoring component (optional, needed for Phoenix scoring)
cd phoenix
pip install -e .

No Dockerfile or Makefile is present; the build relies on Cargo for Rust and pip/uv for Python. Configuration files are not shipped in the repo; the service expects runtime environment variables for database connections, Kafka endpoints, and model paths (these are referenced in thunder/kafka/* and phoenix/run_ranker.rs). Without them the binaries will panic on start.

Real‑World Use

A production feed request hits the home‑mixer HTTP server (home-mixer/server.rs). The server hydrates the user context, calls candidate-pipeline::execute, which pulls recent tweets from Thunder, retrieves candidates from Phoenix, runs the transformer scoring, applies deduplication and relevance filters, then returns the top‑K list to the client. The entire path is bounded to a few hundred milliseconds on typical hardware.

Code Health & Issues

  • High – Add CI – No workflow files; 72 source files are never built automatically.
  • Medium – Enable Dependabot – Only phoenix/pyproject.toml is present; no lockfile or bot config.
  • Medium – Expand Test Suite – 2 test files for 72 source files (≈2.8 % coverage).

Additional observations from the static scan:

  • Deep nesting (max indent 8) in three Rust files makes control flow hard to follow.
  • Repeated 6‑line blocks across 19 hydrator files indicate duplicated logic.

The Bottom Line

The repo delivers a functional, modular For‑You feed pipeline with clear separation between ingestion (Thunder), retrieval (Phoenix) and ranking (home‑mixer). However, the codebase suffers from insufficient automated testing, missing CI/CD, and maintainability concerns (deep nesting and duplicated code). Teams that need a proven baseline for a recommendation stack can adopt it, but should invest early in CI, test expansion, and refactoring of the high‑complexity modules.