nairobi property pricing

The Problem

Real‑estate analysts in Nairobi spend hours manually downloading listings, cleaning price strings, and inferring bedroom counts from noisy titles. The lack of a repeatable pipeline makes cross‑sectional market analysis slow and error‑prone.

What This Does

The repository implements a full ETL chain for Nairobi property data.

• Scraping – nairobi_property_scraper_v2.py and scrape_listings.py drive HTTP calls to sites (e.g., Jiji, Pigiame) and write raw rows to all_raw_listings.csv.
• Cleaning & Enrichment – clean_properties.py, prepare_properties.py, and parser.py normalize price fields, extract bedroom numbers, and compute per‑bedroom affordability. The cleaned output lands in cleaned_properties.csv and location_summary_clean.csv.
• Analysis & Visualization – eda.py produces summary tables and charts under charts/; map_nairobi.py builds the interactive HTML map nairobi_affordability_map.html.

All scripts are pure Python, managed with requirements.txt.

How It Is Wired

Execution starts at the only explicit entry point: the main function in audit_data.py (line 151).

1. main orchestrates the pipeline, calling: audit – validates raw data. clean – invokes clean_properties.clean_properties (the sole function in that module). * detect_unit_scale – determines price units.
2. audit reaches out to the filesystem via save_csv, which creates the target directory (Path(filepath).parent.mkdir). This is the only external write path observed.
3. The most heavily used internal helpers are in scrape_listings.py and nairobi_property_scraper_v2.py: make_record → clean (7 calls) prepares each scraped row. delay, sleep, asleep introduce throttling; they are invoked from 11, 4, and 3 distinct places respectively, giving them the widest “blast radius”. * Network calls occur inside nairobi_property_scraper_v2.py (functions c, row, fetch) and scrape_listings.py (functions scrape_jiji, scrape_pigiame, etc.).
4. The call graph contains 90 resolved edges but no circular imports; each module stands alone, which eases isolated testing.

File‑by‑file responsibilities (derived from the “WHAT EACH FILE IS RESPONSIBLE FOR” block):

No module imports another, so changes to one script rarely ripple through others, but the high nesting depth (up to 12 levels) and broad exception blocks increase cognitive load when modifying the scraper modules.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/nairobi_property_pricing.git
cd nairobi_property_pricing

# Install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

No configuration files or environment variables are referenced in the code; the scripts use hard‑coded URLs and default output paths.

Run the full pipeline:

python audit_data.py

The command executes main, which scrapes, cleans, and writes the final CSVs and charts. Individual stages can be invoked directly (e.g., python nairobi_property_scraper_v2.py to fetch raw listings).

Real‑World Use

A market‑research team could schedule python audit_data.py nightly via cron. The resulting location_summary_clean.csv feeds a BI dashboard, while nairobi_affordability_map.html is embedded in a client‑facing web portal for interactive exploration.

Code Health & Issues

• HIGH – Missing LICENSE – no LICENSE file; default “all rights reserved” blocks reuse.
• HIGH – No CI/CD – repository lacks any GitHub Actions, Travis, etc.; changes are not automatically built or tested.
• HIGH – No build gate for deployable artifacts – charts (e.g., charts/01_top_bottom_prices.png) are produced but never validated by a pipeline.
• MEDIUM – No Dependabot/Renovate – only requirements.txt present; no automated dependency updates.
• MEDIUM – Broad exception handling – nairobi_property_scraper_v2.py and scrape_listings.py use bare except clauses, swallowing errors and complicating debugging.
• MEDIUM – Deep nesting – eda.py, nairobi_property_scraper_v2.py, scrape_listings.py reach 12‑level indentation, making the control flow hard to follow.
• MEDIUM – Oversized file – scrape_listings.py spans 636 lines, a single change can have wide impact.

Additional observations: no test suite, no lockfile (requirements.txt only), and no Dockerfile.

The Bottom Line

The repo delivers a functional end‑to‑end pipeline for Nairobi property data, but the codebase suffers from maintainability issues (deep nesting, large monolithic scripts) and lacks the SDLC scaffolding needed for production confidence (license, CI, tests). It is suitable for internal prototyping or as a foundation to be refactored and hardened before client deployment.