The Problem
Clients need a scalable, data‑driven pipeline that merges “in‑network” (followed accounts) and “out‑of‑network” (global discovery) posts, then ranks them with a single ML model. Maintaining separate hand‑crafted filters and scoring logic is costly and brittle; a unified transformer‑based ranker simplifies feature management while preserving relevance.
What This Does
The repository implements that pipeline in three Rust crates and a Python‑based model server:
home-mixer/ – orchestration layer (home-mixer/main.rs). It hydrates user queries (home-mixer/queryhydrators/.rs), pulls candidates from the two sources (home-mixer/sources/thundersource.rs, home-mixer/sources/phoenixsource.rs), applies a suite of filters (home-mixer/filters/.rs), and finally delegates scoring to the Phoenix transformer (home-mixer/scorers/phoenixscorer.rs). thunder/ – in‑network collector (thunder/main.rs). It ingests tweet events from Kafka (thunder/kafka/.rs) and stores them in a local post store (thunder/posts/poststore.rs). candidate-pipeline/ – a reusable library for candidate handling (candidate-pipeline/lib.rs), exposing types such as Candidate, Scorer, and Selector. phoenix/ – Python package that ships the Grok‑based transformer model (phoenix/recsysmodel.py, phoenix/recsysretrievalmodel.py) and CLI entry points (phoenix/runranker.py, phoenix/runretrieval.py). The pyproject.toml declares the package and its dependencies; a lock file (phoenix/uv.lock) is provided.
Together, the system fetches posts, enriches them with user context, filters out unwanted items (e.g., duplicates, muted keywords), scores each candidate with the model, and returns the top‑K results.
How To Use It
Setup
Rust toolchain (required for all crates) rustup toolchain install stable Build all Rust crates (workspace assumes a root Cargo.toml) cargo build --release
Python environment for Phoenix (uses uv if available)
cd phoenix uv venv # creates .venv uv pip install . # installs the package and its deps
Configuration
Rust – environment variables are read by the services (e.g., KAFKABOOTSTRAPSERVERS, POSTGRESURL). The exact names are referenced in thunder/kafka/utils.rs and home-mixer/server.rs. Python – the model expects a config.yaml (not shipped) that defines feature extraction paths and model checkpoints; phoenix/runranker.py loads it via grok.loadconfig() (see the file for the exact call).
Running
Start the in‑network collector cargo run --bin thunder
Start the orchestrator (home‑mixer) – it will pull from both sources cargo run --bin home-mixer
Run the transformer ranker on a test payload
python -m phoenix.runranker --input testpayload.json
If the repository had a top‑level Cargo.toml defining the workspace, the commands above would resolve the correct binaries; otherwise a manual cargo run --manifest-path home-mixer/Cargo.toml is required.
Real‑World Use
A production service receives a feed request, calls the home-mixer binary with the user ID, and obtains a JSON list of ranked tweet IDs. Internally, home-mixer: Hydrates the request (home-mixer/queryhydrators/userfeaturesqueryhydrator.rs). Pulls in‑network posts from thunder via a gRPC client (code in home-mixer/sources/thundersource.rs). Retrieves out‑of‑network candidates by invoking the Python ranker (phoenix/runretrieval.py) through a subprocess wrapper (home-mixer/sources/phoenixsource.rs). Filters, scores, and selects the top‑K (home-mixer/selectors/topkscoreselector.rs).
The resulting payload is sent back to the front‑end API.
Code Health & Issues
Medium – Missing top‑level Cargo.toml – workspace definition is absent, making builds ambiguous. Low – No CI/CD pipeline – no .github/workflows or other automation; manual builds are the only path. Low – Limited test coverage – only two test files (phoenix/testrecsysmodel.py, phoenix/testrecsysretrievalmodel.py); Rust crates lack unit or integration tests. Low – Documentation gaps – README describes architecture but does not list required env vars or example request payloads. Low – Potential runtime errors – several unwrap() calls in thunder/kafka/.rs could panic on malformed Kafka messages. Low – Dependency lock – a lock file (phoenix/uv.lock) exists, mitigating reproducibility concerns for Python; Rust dependencies rely on Cargo.lock (not listed) – verify its presence.
The Bottom Line
The repo delivers a clear, modular recommendation pipeline that combines Rust‑based ingestion and orchestration with a Python transformer ranker. It is technically sound for teams comfortable managing mixed‑language services, but the lack of a defined Rust workspace, automated CI, and comprehensive tests means additional engineering effort is needed before production deployment. Suitable for organizations that can invest in build‑pipeline setup and add test coverage.