Polymarket data

The Problem

Researchers and analysts need a raw, continuously‑updated record of Polymarket trades, but the blockchain data is large, fragmented, and requires custom cleaning (e.g., unifying token perspectives, filling missing token IDs). Pulling, normalising, and storing this volume manually is error‑prone and time‑consuming.

What This Does

The repository ships a 107 GB, 1.1 billion‑record dataset and a Python toolkit that can fetch on‑chain OrderFilled events, process them into several Parquet tables, and maintain a live sync mode. Core logic lives in polymarket/fetchers/ (RPC calls to Polygon and Gamma API) and polymarket/processors/ (cleaning, token mapping, trade extraction). The CLI in polymarket/cli/main.py exposes commands such as fetch_onchain, process_historical, and update_markets.

How It Is Wired

Execution starts at the console entry point python -m polymarket.cli which runs polymarket/cli/__main__.py → polymarket/cli/main.py. The most used command is cmd_fetch_onchain (line 137) which:

1. Calls get_latest_block → polymarket/fetchers/rpc.py:get_latest_block (uses RPC endpoint from polymarket/config.py:get_rpc_url).
2. Retrieves logs via rpc.get_logs and timestamps via batch_get_timestamps.
3. Writes raw logs with write_batch (defined in polymarket/tools/continuous_fetch.py).

After fetching, cmd_process_historical (line 703) orchestrates the processing pipeline:

• extract_trades (in polymarket/processors/trades.py) parses raw logs, calling load_token_mapping (used 5 times across the repo) and _parse_order_filled.
• Cleaners clean_trades and clean_users (in polymarket/processors/cleaner.py) are invoked, each routing through shared helpers like _process_trades_batch.
• Results are written with write_batch again and progress saved via save_progress.

The most connected modules are polymarket/config (imported by 6 others) and polymarket/fetchers/rpc (imported by 4). Functions with the widest blast radius include extract_trades (6 callers) and load_token_mapping (5 callers). No circular imports are present, so changes in a hub module affect many downstream files but do not create feedback loops.

File‑by‑file responsibility (high‑level):

How To Use It

# Clone the repo
git clone https://github.com/moses-y/Polymarket_data.git
cd Polymarket_data

# Install Python dependencies
pip install -r requirements.txt

# Optional: set RPC endpoint (default in config falls back to env var POLYMARKET_RPC_URL)
export POLYMARKET_RPC_URL="https://polygon-rpc.com"

# Fetch the latest on‑chain logs (writes to data/ folder)
python -m polymarket.cli fetch_onchain

# Process fetched logs into analysis‑ready Parquet tables
python -m polymarket.cli process_historical --start-block 0 --end-block latest

# For continuous sync (runs until stopped)
python -m polymarket.tools.continuous_fetch

The README lists equivalent shell wrappers in scripts/ (e.g., scripts/fetch_onchain.sh), but the Python commands above are the canonical entry points.

Real‑World Use

A quant team could schedule python -m polymarket.cli fetch_onchain every 2 seconds via a cron job, then run process_historical nightly to update trades.parquet and users.parquet. Downstream notebooks would load the Parquet files directly with pandas.read_parquet for factor‑model construction or behavioural clustering.

Code Health & Issues

• High – Cognitive Load – Deep nesting (max depth 8) in fetchers/rpc.py, cli/main.py, processors/cleaner.py. Refactor with early returns or helper functions.
• High – Clarity – Repeated 6‑line blocks across four tools/* scripts. Extract shared helpers to reduce duplication.
• Medium – Resilience – Broad except: clauses in fetchers/rpc.py, cli/main.py, tools/continuous_fetch.py. Replace with specific exception handling.
• Medium – Cognitive Load – cli/main.py is 958 lines; split into sub‑modules (e.g., commands_fetch.py, commands_process.py).
• Medium – SDLC – No test suite present. Adding unit tests for fetchers and processors would catch regressions.
• Medium – CI/CD – No CI configuration; integrating GitHub Actions for linting and testing is advisable.
• Low – Dependency Management – Only requirements.txt, no lockfile; builds may be non‑reproducible across environments.

The Bottom Line

The repo delivers a ready‑to‑use, large‑scale Polymarket dataset and a functional fetch‑process pipeline, but the codebase is tightly coupled, deeply nested, and lacks automated testing or CI. It is suitable for data‑science teams comfortable with Python and willing to invest in refactoring for maintainability.