The Problem

Clients need a quick way to pull the latest headlines from Google News without building a scraper from scratch. Manual browsing or ad‑hoc scripts are fragile, require handling consent cookies, and produce unstructured output.

What This Does

The repository provides a small CLI that fetches a topic‑specific Google News page, extracts article metadata, and writes a CSV file. Core logic lives under src/google_news_scraper/:

  • __main__.py exposes scrape_google_news – the command‑line entry point.
  • scraper.py implements GoogleNewsScraper.scrape, which calls httpx.get to retrieve HTML, then delegates to the parser and cookie helper.
  • cookies.py obtains the consent cookie by launching a headless Chrome driver.
  • parser.py uses GoogleNewsHTMLParser to turn the raw page into Article objects defined in models.py.

Configuration (topic URL building) is in conf.py via GoogleNewsScraperSettings and get_url_for_topic.

How It Is Wired

Execution starts at src/google_news_scraper/__main__.py:19 – the scrape_google_news function. It:

  1. Instantiates GoogleNewsScraper (from scraper.py).
  2. Calls GoogleNewsScraper.scrape(topic_id) – the only function that makes an outbound request (httpx.get).
  3. Inside scrape: get_consent_cookies (from cookies.py) launches Chrome (_init_chrome_driver) and retrieves the consent cookie (_retrieve_consent_cookie_with_driver). httpx.get fetches the topic page. parse (from parser.py) runs _parse_article for each article node, constructing Article dataclasses. _save_to_csv writes the list of Article objects to articles.csv.

Call‑graph summary (14 internal edges):

  • scrape_google_news → GoogleNewsScraper.scrape
  • scrape → get_consent_cookies → _init_chrome_driver / _retrieve_consent_cookie_with_driver
  • scrape → parse → _parse_article → Article
  • scrape → _save_to_csv

No circular imports; each module is isolated, making local changes low‑risk. The widest blast radius is scraper.py because it touches the network, the cookie routine, and the CSV writer.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/google-news-scraper
cd google-news-scraper

# Install dependencies via Poetry (Makefile wraps this)
make install

# Run the scraper for a specific topic
make scrape TOPIC_ID=<topic_id>

make install runs poetry install. make scrape invokes python -m google_news_scraper <TOPIC_ID> (implemented in __main__.py).

The tool expects a valid Google News topic ID (the string after /topics/ in the URL). Output appears as articles.csv in the current directory.

Real‑World Use

A data‑engineer can embed the CLI in a nightly ETL job:

#!/usr/bin/env bash
TOPIC=CAAqJggKIiBDQkFTRWdvSUwyMHZNRGx6TVdZU0FtVnVHZ0pWVXlnQVAB
cd /opt/google-news-scraper && make scrape TOPIC_ID=$TOPIC
aws s3 cp articles.csv s3://my-bucket/news/$(date +%F).csv

The CSV is then consumed by downstream analytics pipelines.

Code Health & Issues

  • High – No LICENSE – repository root lacks a licence file; redistribution rights are undefined.
  • High – No CI/CD – no workflow files; changes are not automatically built or tested.
  • High – No Dependabot – only pyproject.toml present; automatic dependency updates are missing.
  • Medium – Broad exception handling – src/google_news_scraper/cookies.py uses a bare except: that swallows all errors.
  • Medium – No tests – the repo contains no tests/ directory or test files.

All findings come from deterministic static analysis; no additional issues were inferred.

The Bottom Line

The project delivers a functional, single‑purpose Google News scraper with clear module separation and a simple CLI. It is usable as‑is for small‑scale data pulls but lacks production safeguards: no license, CI, or test suite, and it contains a risky broad exception catch. Teams that need a quick prototype will benefit; larger deployments should add licensing, testing, and tighter error handling before integration.