The Problem

This repository analyzes AI exposure across 342 US occupations using Bureau of Labor Statistics data, but lacks automation guardrails. A consultant would find no CI/CD pipeline, no dependency lockfile, and no license — meaning changes merge unverified, builds are non-reproducible, and reuse rights are undefined.

What This Does

The code implements a six-stage data pipeline across 8 Python source files. scrape.py drives Playwright to download raw BLS occupation HTML into html/ (342 files). parse_detail.py and process.py use BeautifulSoup to convert those into clean Markdown in pages/. make_csv.py extracts structured fields (pay, education, job count, growth outlook, SOC code) into occupations.csv. score.py sends each occupation's Markdown to an LLM (Gemini Flash via OpenRouter) with a scoring rubric, producing scores.json. build_site_data.py merges CSV stats and AI exposure scores into compact site/data.json for the frontend. The entry point is site/index.html, an interactive treemap where rectangle area = employment and color = AI exposure (green to red). A prompt.md file (~45K tokens) aggregates all data for LLM analysis.

How It Is Wired

Execution starts at the Playwright scraper (scrape.py), which writes 342 HTML files into html/. Those feed parse_detail.py (indentation depth 9, 3 levels of nesting) which converts HTML to Markdown in pages/. process.py (1 branch point) then processes those Markdown files. The scored output flows through score.py (56 branch points over 154 lines) to scores.json, which build_site_data.py merges with occupations.csv to produce site/data.json. The module graph shows scrape as the root hub (Ca=1, Ce=0) — every other module depends on it indirectly. make_csv and build_site_data are leaf nodes with no outgoing imports. The deepest blast radius resides in score.py: it imports no other modules (Ce=0) but is imported by no other file (Ca=0), making it a standalone scorer with no downstream dependents, yet its LLM call leaves the process boundary. parse_detail.py and `make_csv.py each carry the high cognitive-load finding of deep nesting (x3), requiring flattening via guard clauses.

How To Use It

Setup: No lockfile exists, but pyproject.toml declares 4 dependencies. Install with:

pip install -r requirements.txt  # if present, otherwise pip install . per pyproject.toml

No requirements.txt or explicit install_requires in the measured artifacts was confirmed; verify the pyproject.toml manifest before installing.

Configuration: No environment variables or API keys are documented. score.py requires an OpenRouter API key for the Gemini Flash LLM — source this and place it in the runtime environment or a .env file the script reads.

Running it: The pipeline has no single entry point. To rebuild from scratch, run sequentially:

python scrape.py      # populates html/
python parse_detail.py  # produces pages/
python make_csv.py    # produces occupations.csv
python score.py       # produces scores.json
python build_site_data.py  # produces site/data.json

Then open site/index.html in a browser.

Real-World Use

A data journalist could clone this repo, run the pipeline to generate fresh scores.json, then embed site/data.json into a client-facing dashboard. The treemap visualization (site/index.html) immediately shows which occupational clusters (by BLS category) face the highest AI exposure, weighted by employment — useful for workforce planning or policy briefings. The prompt.md file provides a ready-made, data-grounded context for ad-hoc LLM analysis of exposure trends.

Code Health & Issues

  • [HIGH] Add a LICENSE — no licence file at the repository root; redistribution rights are undefined. Without a license the default is all rights reserved, so this code cannot legally be reused in client engagements.
  • [HIGH] Add CI/CD workflow — 8 source files, no CI configuration; every change merges with nobody having run the build once.
  • [MEDIUM] Enable Dependabot or Renovate — 1 manifest (pyproject.toml), no update bot configured; published advisories sit unpatched until manual audit.
  • [MEDIUM] Deep nesting x3 — parse_detail.py, make_csv.py, score.py each have indentation depth up to 9; control flow is hard to follow. Fix: flatten with early returns/guard clauses.
  • [MEDIUM] High branching density x2 — parse_detail.py and make_prompt.py have 56 branch points over 154 lines; decompose decision-heavy logic.

The Bottom Line

The pipeline is functional and produces a compelling treemap visualization of AI exposure across the US job market, but it operates without a license, test coverage, or automation gates. A consultant would accept the data pipeline as-is for ad-hoc analysis but would insist on a license, CI, and a lockfile before recommending this code for any production or shared estate. The scoring logic and LLM integration are the strongest technical assets; the surrounding SDLC gaps are the primary blockers to adoption.