The Problem

Businesses that need large‑scale lead lists must collect name, address, phone, website, rating, reviews, coordinates, and often email from Google Maps. Doing this manually is slow, error‑prone, and violates Google’s UI limits. A reliable scraper that can run headless or via an API is needed to automate the extraction.

What This Does

The repo is a portfolio of 12 small Go projects that together implement a Google Maps scraper with CLI, Web UI, and serverless options.

  • gmaps/ holds the core scraping logic – entry.go coordinates jobs, job.go builds request payloads, place.go parses place details, and reviews.go walks review pages.
  • runner/ provides concrete runners: webrunner/webrunner.go serves the UI, databaserunner/databaserunner.go writes to PostgreSQL, filerunner/filerunner.go stores JSON/CSV files, and lambdaaws/ packages the scraper for AWS Lambda.
  • postgres/ (provider.go, resultwriter.go) abstracts DB connections and bulk inserts.
  • deduper/ (deduper.go, hashmap.go) removes duplicate leads before export.
  • web/ contains static assets and the HTML template web/static/templates/index.html for the UI.
  • cmd/tools/tools.go and main.go expose a CLI (go run main.go …) that selects a runner based on flags.

How It Is Wired

Execution starts in main.go. It parses command‑line arguments (e.g., --mode=web, --output=postgres) and creates a runner from runner/runner.go.

  • Web mode → runner/webrunner/webrunner.go starts an HTTP server, serves index.html, and on a scrape request calls gmaps/entry.go → Entry.Start() → creates a Job (gmaps/job.go) → invokes searchJob, emailJob, and reviewsJob. Each job uses the rod‑based Chrome driver (Dockerfile.rod) to load Google Maps pages, extracts JSON blobs, and fills a Place struct (place.go).
  • Database mode → runner/databaserunner/databaserunner.go receives the Place slice, calls postgres/provider.go to open a connection, then postgres/resultwriter.go performs bulk INSERTs.
  • File mode → runner/filerunner/filerunner.go marshals the same structs to CSV/JSON files.

All runners share the deduper (deduper/deduper.go) to drop duplicate lead hashes before any output. The leadsdb/leadsdb.go module can push results to the external LeadsDB API, but is optional.

The only hub with a large blast radius is gmaps/entry.go (661 LOC) and gmaps/reviews.go (deep nesting, 9‑level indentation). Calls from these files cascade to every runner, so changes here affect the whole suite. No circular imports were detected.

How To Use It

# Clone the repository
git clone https://github.com/moses-y/google-maps-scraper
cd google-maps-scraper

# Build the container (Dockerfile uses ubuntu:20.04 + golang:1.25.6-trixie)
docker build -t gmaps-scraper .

# Run the CLI (example: scrape “coffee shops” in “Berlin” and write CSV)
docker run --rm gmaps-scraper go run main.go \
    --query="coffee shops" --location="Berlin" \
    --output=file --format=csv

For the Web UI replace --output=file with --output=web; the server starts on port 8080 and serves web/static/templates/index.html.

Configuration files are not shipped; any required Google Maps API keys or proxy settings must be supplied via environment variables referenced in the code (e.g., PROXY_URL). The repository includes a Makefile and docker-compose.dev.yaml for local development, but specific targets are not enumerated here.

Real‑World Use

A SaaS lead‑generation platform can embed this scraper as a microservice. The platform launches the Docker image with a job definition (query + location) via the CLI, streams the resulting CSV into an S3 bucket (runner/filerunner), and then triggers a downstream ETL pipeline. The same binary can be deployed as an AWS Lambda (runner/lambdaaws) for on‑demand scrapes, reusing the identical core logic.

Code Health & Issues

  • High – GitHub Actions are pinned to tags only (docker/login-action@v3, etc.). Pin to commit SHA to avoid supply‑chain drift.
  • Medium – No Dependabot/Renovate config; add .github/dependabot.yml.
  • Medium – Base images in Dockerfile are mutable tags (ubuntu:20.04, golang:1.25.6-trixie). Pin by digest.
  • Medium – No dependency‑vulnerability scan in CI; add dependency-review-action or osv-scanner.
  • Medium – Checkout step keeps the token; set persist-credentials: false.
  • Medium – Container runs as root; create a non‑root USER in Dockerfile.
  • Medium – Test coverage is low (1 test file for 35 source files). Add unit tests for high‑fan‑in modules (gmaps/entry.go, runner/).
  • Low – Workflow jobs lack timeout-minutes; define reasonable limits.

Additional observations: the repository includes a licence (MIT) and a .github/FUNDING.yaml. No secrets were found in the history. Documentation is limited to README.md and a few markdown guides; usage instructions are sparse, requiring the consumer to infer CLI flags.

The Bottom Line

The repo provides a functional, multi‑mode Google Maps scraper built in Go, with clear separation of core scraping, runners, and output adapters. However, deep nesting and oversized files make maintenance harder, and the CI/CD pipeline lacks essential hardening (pinned actions, vulnerability scanning, non‑root containers). It is suitable for teams comfortable navigating Go codebases and willing to add tests and CI safeguards before production use.