The Problem

Manually checking Y Combinator's job board is tedious and easy to forget. This scraper automates the daily check and pushes new listings to WhatsApp, so a job seeker gets notified without opening the browser.

What This Does

The repo is a small Python project that scrapes Y Combinator job listings, stores them in a local SQLite database, and sends a WhatsApp alert for new jobs. It runs on a schedule via GitHub Actions.

The code is split into four modules under src/: scraper.py handles fetching and parsing job pages, database.py manages SQLite storage, messaging.py sends WhatsApp messages through Twilio, and main.py is the entry point that orchestrates the flow. Tests live in tests/ and cover the scraper, database, and messaging logic.

How It Is Wired

Execution starts at main in src/main.py:9. It calls init_db to create the database, get_existing_job_urls to load already-seen jobs, scrape_jobs to fetch new listings, insert_job for each new one, and send_whatsapp_message to alert the user. The path from entry point to external effect is short: main -> scrape_jobs -> fetch_page_content -> setup_driver (Selenium), then main -> insert_job -> get_db_connection (SQLite), and main -> send_whatsapp_message (Twilio API).

The most connected modules are src/database.py, src/messaging.py, and src/scraper.py, each with one importer and no imports of their own — they are leaf modules with zero instability. get_db_connection is called from 3 places, making it the highest-blast-radius function; changing its signature breaks init_db, insert_job, and get_existing_job_urls. parse_jobs calls clean_text three times per run, so any change to text normalization affects every parsed listing. src/sandbox_reminder.py is a separate script with its own main and setup_driver, not wired into the primary flow.

The module graph shows no circular dependencies. The main risk for a contributor is the tight coupling between scraper.py and database.py — both must change together if the job schema evolves.

How To Use It

Setup:

git clone https://github.com/moses-y/ycombinator-job-scraper
cd ycombinator-job-scraper
pip install -r requirements.txt

Configuration: Set four environment variables: TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_PHONE_NUMBER, and YOUR_PHONE_NUMBER. The README documents these; there is no .env file or config loader, so export them in your shell or CI environment.

Running it:

python src/main.py

The GitHub Actions workflow at .github/workflows/scraper.yml schedules this daily. Note there is no lockfile (requirements.txt only), so builds are not fully reproducible.

Real-World Use

A job seeker sets up the scraper once, configures Twilio, and lets the daily cron job run. Each morning, new YC job postings arrive as WhatsApp messages. The SQLite database prevents duplicate alerts across runs by storing seen URLs in get_existing_job_urls.

Code Health & Issues

Static analysis (measured, not opinion) found:

  • Low - Dependencies declared without a lockfile - requirements.txt - non-reproducible builds; pin exact versions or add a lockfile.

SDLC observations from the file structure:

  • Med - No Dockerfile - setup requires a local Python environment with ChromeDriver; the assets/chromedriver-win64/ binary is Windows-only, so the scraper will not run on Linux CI without a separate driver install.
  • Low - Committed binary - assets/chromedriver-win64/chromedriver.exe is checked into the repo, which bloats the repository and risks platform mismatch.

The Bottom Line

This is a focused, working scraper that solves a real problem for a single user. The code is small and readable, with tests and CI in place. The main weaknesses are the missing lockfile and the Windows-only ChromeDriver, which will bite anyone trying to run it on a different platform. It is suitable for personal use or as a template for a notification-driven scraper.