The Problem

Hiring managers and candidates lack a single, data‑driven view of AI engineering roles, required skills, and interview expectations. Existing resources are fragmented, often anecdotal, and hard to keep current.

What This Does

The repository aggregates 6,964 real job descriptions (YAML files in job‑market/data_structured/*) and interview narratives (Markdown in interview/data/*). It ships analysis scripts that turn raw postings into skill heat‑maps, trend charts, and role archetypes (job‑market/_internal/analysis/*). The interview/_internal/main.py entry point orchestrates job‑post fetching from Reddit (fetch_reddit.py) and a specialized XAI search (xai_search.py) to keep the dataset fresh.

How It Is Wired

python interview/_internal/main.py starts the process.

  1. main.py imports and calls fetch_reddit() from fetch_reddit.py, which scrapes Reddit threads to harvest new job postings.
  2. The results are passed to xai_search() in xai_search.py, a lightweight wrapper around a vector‑store query that validates the postings against the existing dataset.
  3. Validated entries are written as YAML files under job‑market/data_structured/*.
  4. The same main.py can trigger the analytical pipeline by importing job‑market/_internal/analysis/analyze.py, which in turn uses helper modules such as canonicalize_skills.py, skills_analysis.py, and trends.py to generate CSV exports and visualizations.

The control flow is linear: entry point → fetch → validate → persist → analyze. There are no hidden side‑effects; filesystem writes are the only external impact, located in the YAML folders.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/ai-engineering-field-guide
cd ai-engineering-field-guide

# Install deps (uv is recommended; pip works too)
uv pip install -r interview/_internal/pyproject.toml
# or
pip install -r interview/_internal/pyproject.toml

# Run the fetch & analysis pipeline
python interview/_internal/main.py

The script reads no external config files; all hard‑coded URLs and API keys are stored in environment variables (e.g., REDDIT_TOKEN, VECTOR_STORE_ENDPOINT). If absent, the script logs a warning and skips the corresponding step.

Real‑World Use

A hiring team can execute the pipeline nightly to generate an updated job‑market/_internal/analysis/trends.csv. The CSV can be fed into a BI tool or a custom dashboard to surface the latest skill gaps and hiring velocity for each city.

Code Health & Issues

  • Medium – No CI/CD – The repo lacks a .github/workflows/* or similar build trigger.
  • Medium – No LICENSE – Redistribution rights are unclear.
  • Low – Missing lockfile – pyproject.toml declares dependencies without a poetry.lock or Pipfile.lock, making builds non‑reproducible.
  • Low – Tests not integrated – 290 test files exist but are not run automatically.

The Bottom Line

The guide offers a rich, well‑structured dataset and straightforward analysis tools for AI engineering hiring research. Its lack of CI, license, and dependency locking limits production readiness, but it is valuable for researchers, recruiters, and candidates who need up‑to‑date, data‑driven insights.