The Problem

Business development teams often need hundreds of contact records (phone, email, address) for a specific service in a target city. Manually extracting that data from Google Maps, Yelp or similar sites is slow, error‑prone, and scales poorly.

What This Does

The repository bundles a collection of scrapers that automate the extraction of lead data from Google Maps and Yelp, then write the results to CSV. The core logic lives under py_lead_generation/src/engines/ – e.g. google_maps/engine.py and yelp/engine.py – while a legacy Flask UI lives in archived/app/. The top‑level run.py provides a simple CLI entry point that imports the engines, runs them asynchronously, and calls the save_to_csv method on each engine instance.

Key files:

  • run.py – entry script used in the README quick‑start.
  • py_lead_generation/src/engines/base.py – abstract base class that defines run, save_to_csv, and helper _open_url_and_wait.
  • py_lead_generation/src/google_maps/engine.py – concrete implementation for Google Maps.
  • py_lead_generation/src/yelp/engine.py – concrete implementation for Yelp.
  • archived/app/views.py – Flask view layer (login, subscription checks, DB writes).

How It Is Wired

Execution begins at run.py:5 (main). The call flow is:

  1. main → GoogleMapsEngine(...).run() (via base.run).
  2. Inside base.run, the engine calls _open_url_and_wait (2 places) which launches a headless browser, navigates to the search URL, and waits for page load.
  3. The browser actions invoke search → _locate → _scrape (in archived/google-maps/extractor.py).
  4. _scrape eventually calls dump (from archived/bufferization.py) which writes raw HTML to a temporary buffer.
  5. The scraper extracts JSON payloads and makes outbound HTTP calls (requests.get) – the only network‑leaving edge detected.
  6. After the async run completes, engine.save_to_csv() (implemented in py_lead_generation/src/misc/writer.py → CsvWriter.append) writes the parsed rows to a CSV file on disk.

A parallel path runs YelpEngine(...).run() with the same internal sequence, using its own search implementation under archived/yellow-pages/extractor.py.

The Flask side (archived/app/app.py) is not part of the CLI flow; it starts a web server that imports archived/app/views.py. That module defines 13 functions (including login_required, subscription_required) which read/write the database (archived/app/models.py) and also perform network calls (e.g., sending emails via archived/emails/hunterio.py).

The internal import graph shows 40 Python modules with only 10 import edges and no circular dependencies, indicating a relatively flat structure. The most fan‑in modules are py_lead_generation/src/engines/base.py (6 imports/exports) and archived/app/views.py (13 functions, heavy DB/network impact).

How To Use It

# 1. Install the published package (recommended)
pip install py-lead-generation

# 2. Or clone the source
git clone https://github.com/moses-y/Lead-Generation
cd Lead-Generation

# 3. Install runtime dependencies
pip install -r py_lead_generation/requirements.txt

# 4. Run the example CLI
python run.py

The script will prompt for a query and location, then generate two CSV files (google_maps_results.csv, yelp_results.csv). If you need the legacy Flask UI, start it with:

cd archived
export FLASK_APP=app/app.py
flask run

Configuration for the Flask app lives in archived/app/config.py (DEBUG flag, SECRET_KEY). No other environment variables are required for the CLI path.

Real‑World Use

A sales‑ops engineer can embed the engines in an internal pipeline:

from py_lead_generation import GoogleMapsEngine, YelpEngine

gm = GoogleMapsEngine("plumber", "Austin, TX")
yelp = YelpEngine("plumber", "Austin, TX")

await asyncio.gather(gm.run(), yelp.run())
gm.save_to_csv("gm_leads.csv")
yelp.save_to_csv("yelp_leads.csv")

The resulting CSVs feed directly into a CRM import or a downstream validation service.

Code Health & Issues

  • High – Debug mode enabled – archived/app/config.py sets DEBUG = True.
  • High – Hard‑coded secret key – archived/app/config.py contains a 128‑character fallback SECRET_KEY.
  • Medium – Dependabot missing – No automated dependency update bot configured.
  • Medium – No dependency‑vulnerability scan in CI – .github/workflows/python-publish.yml lacks a scan step.
  • Medium – Checkout persists token – persist-credentials not disabled in the workflow.
  • Medium – Test coverage low – Only 1 test file for ~40 source files (≈2 % coverage).
  • Low – Workflow jobs lack timeout – No timeout-minutes set, risking long‑running runs.

Additional observations: the repository ships a requirements.txt without a lockfile, making reproducible builds harder; the Flask app’s config is the only place where environment variables are expected, but the CLI path does not use any. Documentation is limited to the README; no pyproject.toml or pre‑commit hooks are present.

The Bottom Line

The project delivers functional, asynchronous scrapers for Google Maps and Yelp that can be invoked via a simple CLI, but the codebase shows limited test coverage, insecure defaults in the Flask config, and missing CI safeguards. It is suitable for prototyping or internal tooling where the security trade‑offs are managed, but production deployments should first address the high‑severity config issues and add automated testing.