The Problem

Extracting structured data from news websites is a fragmented problem: crawling, HTML parsing, boilerplate removal, and date/author extraction each need different tools, and gluing them together is most of the work. news-please packages that pipeline into one installable Python package, so a developer can go from a URL to a structured article record without assembling five libraries.

What This Does

news-please is a news crawler and extractor with two modes. The library mode (newsplease/NewsArticle.py) exposes from_url, from_urls, and from_file for one-off extraction. The CLI mode (newsplease/__main__.py) runs continuous crawling via RSS, sitemaps, or recursive link-following, and writes results to JSON, PostgreSQL, ElasticSearch, or Redis.

The extraction pipeline lives in newsplease/pipeline/. It runs multiple extractors (newspaper_extractor.py, readability_extractor.py, date_extractor.py, lang_detect_extractor.py) and a comparer/ module that merges their outputs by field—title, author, date, text—into a single NewsArticle. A commoncrawl_crawler.py module pulls from Common Crawl's WARC archives.

How It Is Wired

Execution starts at main in newsplease/__main__.py:754, which reaches 84 functions—the widest blast radius in the repo. It parses CLI args, sets up logging, and dispatches to crawler management functions. The shortest path to an external effect is main -> __setup__, which calls os.makedirs on the filesystem.

Two files carry most of the weight:

  • newsplease/__main__.py — 30 functions, 3 classes, called from 12 files. Handles CLI, daemon mode, and crawler lifecycle.
  • newsplease/pipeline/pipelines.py — 41 functions, 16 classes. The Scrapy item pipeline: process_item calls extract_relevant_info, which is the hub for extraction.

Key routing functions: get_allowed_domain (called from 9 places) and extract (8 places) in url_extractor.py and article_extractor.py respectively. __init__ calls section 33 times—a sign of heavy configuration-driven branching.

The repo touches the outside world in 4 places: 4 functions do database I/O, 12 read/write files, 8 make network calls, 4 do crypto (likely MD5 for filename hashing in savepath_parser.py). The module graph has no cycles, which keeps changes contained; the most unstable module is comparer.py (instability 0.89), meaning it depends on many things but few depend on it—safe to refactor.

How To Use It

pip install news-please

Library mode:

from newsplease import NewsPlease
article = NewsPlease.from_url('https://example.com/article')
print(article.title)

CLI mode: run python -m newsplease (the __main__.py entry point). Configuration lives in newsplease/config/config.cfg and sitelist.hjson. The Dockerfile builds a container image; docker.sh wraps the build/run.

Real-World Use

A media-monitoring service that ingests articles from a publisher list: a scheduler calls NewsPlease.from_urls([...]) daily, stores results in PostgreSQL via the pipeline's storage backends, and the commoncrawl.py example pulls historical archives for backfill.

Code Health & Issues

Static analysis found 34 issues (18 high, 16 medium). The dominant problem is deep nesting (22 instances, max depth 9) in url_extractor.py, config.py, and savepath_parser.py—control flow is hard to follow. Duplicated code appears as 41 repeated 6-line blocks across 10 files, mostly in the crawler modules. Broad exception handling in __init__.py and the commoncrawl modules swallows errors. pipelines.py and __main__.py are oversized (848 lines combined) with high branching density.

SDLC gaps: no test suite, no CI/CD, and no dependency lockfile. The Dockerfile uses python:3.6.5-alpine3.7—an old, unpinned base image, with no non-root USER directive. The README claims Python 3.8+ support, which contradicts the Dockerfile's 3.6 image.

The Bottom Line

A genuinely useful tool with solid architecture (clean module graph, no cycles) but aging code quality and zero automated verification. The extraction pipeline is the valuable part; the CLI and crawler layers show their age. Use it as a library and add your own tests before depending on it for production.