The Problem
Collecting and normalising news articles from heterogeneous sites is labor‑intensive. Each outlet uses a different HTML layout, RSS format, or sitemap structure, and most open‑source crawlers handle only a subset of these patterns. Teams that need a reliable, repeatable feed of headline, body, author, date, language and image must either write site‑specific scrapers or stitch together multiple tools.
What This Does
news-please bundles a Scrapy‑based crawler with a set of extractors that wrap newspaper4k, readability, and custom heuristics. The core entry point is newsplease/main.py, which parses CLI arguments and builds a Scrapy CrawlerProcess.
Key modules: newsplease/crawler/spiders/ – concrete spiders (newspleasespider.py, rsscrawler.py, sitemapcrawler.py) that drive URL discovery. newsplease/pipeline/extractor/ – extractor classes (newspaperextractor.py, readabilityextractor.py, langdetectextractor.py) that populate a NewsArticle object (newsplease/NewsArticle.py). newsplease/config/ – default configuration files (config.cfg, configlib.cfg, sitelist.hjson) loaded by newsplease/config.py.
The library mode is exposed via newsplease/init.py where NewsPlease.fromurl() returns a fully populated NewsArticle instance.
How To Use It
Setup
Install from PyPI (recommended) pip install news-please
Or build the Docker image provided in the repo docker build -t news-please .
Both paths rely on requirements.txt for Python dependencies; the Dockerfile copies the source and runs pip install -r requirements.txt.
Configuration
The default behaviour reads newsplease/config/config.cfg. Users can supply an alternative config file with the --config CLI flag (see newsplease/main.py argument parser). No secret keys are required for the core crawler; optional storage back‑ends (PostgreSQL, Elasticsearch, Redis) are configured in the same file.
Running
CLI mode (crawl a site recursively): python -m newsplease -u https://example.com --output json
The -u/--url flag is defined in newsplease/main.py; --output selects the writer (JSON, PostgreSQL, etc.). Library mode (single URL extraction): from newsplease import NewsPlease article = NewsPlease.fromurl('https://www.nytimes.com/2023/01/01/world/europe/article.html') print(article.title, article.publish_date) CommonCrawl archive (bulk download): python -m newsplease.examples.commoncrawl \ --output-dir ./cc-data \ --date-from 2022-01-01 --date-to 2022-12-31
Parameters are parsed in newsplease/examples/commoncrawl.py.
Real‑World Use
A media‑monitoring service can schedule nightly runs of news-please against a list of publisher roots, store the JSON output in an S3 bucket, and ingest the data into a downstream analytics pipeline. Example workflow:
import json, pathlib, subprocess
publishers = ['https://www.bbc.com', 'https://www.cnn.com'] for p in publishers: subprocess.run(['python', '-m', 'newsplease', '-u', p, '--output', 'json', '--out-dir', f'./out/{pathlib.Path(p).netloc}'])
The resulting files contain a uniform schema ready for indexing in Elasticsearch or for sentiment analysis.
Code Health & Issues
Med – No test suite – No tests/ directory or pytest files; core paths are unverified. Med – Missing CI/CD – .github contains issue templates and funding config but no workflow YAML; builds are not automatically validated. Low – No lockfile – Dependencies are listed only in requirements.txt; reproducible installs depend on external package versions. Low – Config files in source – config.cfg and sitelist.hjson are committed with default values; users must edit them for production storage credentials. Low – Limited documentation – README covers basic CLI usage; advanced storage options lack examples.
No obvious security secrets are present, and the Dockerfile follows standard best practices (non‑root user, explicit COPY).
The Bottom Line
news-please delivers a functional, out‑of‑the‑box solution for crawling and extracting structured news content, with both CLI and library interfaces. It is suitable for prototypes, research projects, or small‑to‑medium pipelines where rapid setup outweighs the need for a comprehensive test suite or CI pipeline. For production‑grade deployments, teams should add automated tests, lock dependencies, and integrate CI to mitigate the current gaps.