AI Exposure of the US Job Market
The Problem
Everyone’s talking about how AI is going to steal jobs, but no one has a clear picture of which occupations are actually at risk — or why. This repo tackles that problem by quantifying “AI exposure” for 342 occupations in the US economy using data scraped from the Bureau of Labor Statistics (BLS). If you’ve ever wondered whether accountants or plumbers should start panicking, this project has some answers.
What This Does
At its core, this repo is about taking government data (the BLS Occupational Outlook Handbook) and turning it into something useful. Here's the rundown: Scrape: scrape.py uses Playwright (non-headless mode because BLS blocks bots) to grab 342 HTML pages of job descriptions. These are dumped into the html/ folder — yes, all 196 files are raw HTML. Parse: parsedetail.py and process.py employ BeautifulSoup to clean up the HTML and spit out Markdown files (in pages/) with structured data fields. Tabulate and Score: makecsv.py creates a tidy occupations.csv, summarizing pay, job count, growth projections, etc. Then, score.py sends the Markdown descriptions to an LLM (Gemini Flash via OpenRouter) for AI Exposure scoring (0-10). Results land in scores.json. Build the Treemap: buildsitedata.py smooshes the occupations.csv and scores.json into site/data.json for the frontend, which is a pretty cool treemap visualization (check it out here).
Real-World Use
Say you’re a career counselor, worried about how emerging tech will impact your clients. You could clone this repo, tweak score.py to use a different LLM tailored to your needs (ChatGPT, Claude, whatever), and rerun it with updated job descriptions from next year’s BLS data. Then, plug the scores into the treemap (site/index.html) for a visual tool to guide discussions on career planning.
Want a quick snapshot? Just open the treemap. Curious about how scores are calculated? Dive into the prompt.md file for the LLM rubric and methodology.
The Bottom Line
This is a decent project for anyone who’s into data-driven career analysis or just wants to geek out over AI’s impact on jobs. The data pipeline is solid, but unless you’re customizing it or updating the dataset, you’re mostly here for the treemap. It’s not groundbreaking, but it’s a fun and informative way to slice and dice BLS data. If you're scared of AI taking your job, maybe spend less time worrying and more time on score.py.