The Problem

Online‑marketing teams routinely wrestle with URL lists, keyword explosions, and SERP metadata. Without dedicated tooling, that work devolves into manual spreadsheet gymnastics or fragile custom scripts. advertools aims to replace that ad‑hoc pipeline with a small, import‑able Python package, but its own code base bears structural debts that make extension and refactoring harder than necessary.

What This Does

advertools is a collection of independent functions for crawling, SERP extraction, keyword generation, stop‑word filtering, and more. Key modules and their responsibilities (from the responsibility map) include:

  • advertools/spider.py – core crawler logic (_crawl_or_not, _extract_images, matches_any, get_max_cmd_len).
  • advertools/serp.py – SERP‑scraping and analytics (the largest file at ~2105 lines, high ripple risk).
  • advertools/kw_generate.py – keyword combinatorics and match‑type handling.
  • advertools/extract.py – generic text extraction helpers.
  • advertools/cli.py – command‑line entry point with _make_headline, _split_options, _format_df.

Functions such as _dict_product, _json_to_df, and ad_create are the most frequently invoked across the code base, and many return DataFrames that integrate directly with pandas.

How It Is Wired

  • Entry point: advertools/cli.py (and the setup.py/pyproject.toml‑driven pip install advertools).
  • Import graph: 62 files analysed; 15 modules live in circular dependencies, the primary cycle involving advertools/__init__.py, advertools/spider.py, and advertools/ad_create.py.
  • Hub module: advertools/__init__.py is imported by 19 other modules; changes here have a blast‑radius effect.
  • Oversized files: advertools/serp.py (2105 loc), advertools/spider.py, and advertools/stopwords.py exceed 1000 lines, making control flow hard to follow (max indentation depth 8).
  • Duplicated logic: 56 repeated 6‑line blocks appear across 11 files (advertools/_yt_helpers.py, advertools/serp.py, advertools/ad_create.py, advertools/reverse_dns_lookup.py, …).
  • Exception handling: bare except clauses in advertools/serp.py, advertools/sitemaps.py, and advertools/_regex_helpers.py swallow errors indiscriminately.

Outside the package, the code touches the filesystem (log parsing, sitemap XML), makes HTTP requests (SERP crawling, Twitter API), and produces DataFrames consumable by pandas or downstream ML tooling.

How To Use It

Setup

python3 -m pip install advertools

No lockfile is committed; reproducible builds require adding one (e.g., pip-compile from pyproject.toml).

Configuration

  • Basic functions need no keys.
  • Twitter‑related calls (advertools/twitter.py) require a bearer token; set AD_TWITTER_BEARER_TOKEN in the environment or a .env file.
  • Claude‑enhanced SERP (advertools/serp_claude.py) expects an API key; the README references the readthedocs page for details.

Running it

# List top keywords for a domain
python3 -m advertools.kw_generate --domain example.com --max 500

# Quick crawl of a site
python3 -m advertools.spider --seed https://example.com --max_pages 100

# CLI help
python3 -m advertools.cli --help

Real‑World Use

A marketer wants to audit a competitor’s robots.txt, pull the first 200 SERP results for “best running shoes”, and export the keyword opportunities to a CSV for further analysis with pandas. A three‑line workflow using the package:

import advertools as at

robots = at.robotstxt("https://competitor.com/robots.txt")
serp   = at.serp("best running shoes", limit=200)
keywords = at.kw_generate(domain="competitor.com", max=500)
keywords.to_csv("opp_keywords.csv", index=False)

The resulting keywords DataFrame can be joined with serp results or fed into any downstream model.

Code Health & Issues

  • Import cycles (High) – 15 modules in a circular dependency; advertools/__init__.py, advertools/spider.py, advertools/ad_create.py are members. Break the cycle by extracting shared types or deferring imports.
  • Oversized files (High) – advertools/serp.py (2105 loc), advertools/spider.py, advertools/stopwords.py. Split by responsibility (e.g., separate parsing from analysis).
  • Duplicated code blocks (Medium) – 56 repeated 6‑line snippets across 11 files; extract a shared helper (advertools/_regex_helpers.py or a new advertools/helpers.py).
  • Broad exception handling (Medium) – bare except in advertools/serp.py, advertools/sitemaps.py, advertools/_regex_helpers.py; replace with specific exception types.
  • Deep nesting (High) – max indentation depth 8 in advertools/spider.py, advertools/regex.py, advertools/serp_claude.py; flatten with guard clauses.
  • Hub module (Medium) – advertools/__init__.py imported by 19 modules; keep it stable and move volatile logic out.

No lockfile is committed (pyproject.toml declares dependencies without a requirements.txt lock), which can cause non‑reproducible builds; add a lockfile or use uv pip compile.

The Bottom Line

advertools delivers a pragmatic set of productivity tools for online‑marketing data wrangling, with many functions that return ready‑to‑use DataFrames and a CLI that lowers the barrier to entry. However, the code base suffers from import cycles, oversized files, and duplicated logic that increase cognitive load and risk for future changes. Teams that need quick URL‑level analysis, keyword generation, or SERP extraction will find immediate value, but they should plan to refactor the identified hot‑spots to keep the package maintainable as it grows.