The Problem

Cryptocurrency market data is only useful when it flows continuously from source to analytics. Manual batch pulls give stale prices, and ad-hoc scripts don't survive schema changes or service restarts. This repo addresses that by wiring CoinPaprika's live ticker API through a full streaming pipeline: PostgreSQL for staging, Kafka for transport, Spark for transformation, and Cassandra for queryable storage.

What This Does

The pipeline is three Python modules orchestrated as a streaming ETL. etl_coin_paprika.py defines an Airflow DAG with etl_extract_data and etl_load_data functions that pull ticker data from CoinPaprika and upsert it into PostgreSQL. coin_paprika_producer.py has a single fetch_postgres function that reads new rows and publishes them to a Kafka topic. spark_cassandra_etl.py consumes that topic, parses the JSON payloads, computes derived fields like market capitalization, and writes to Cassandra.

The pyproject.toml declares dependencies; .python-version pins the interpreter. There is no Dockerfile, no test suite, and no CI configuration.

How It Is Wired

Execution starts in etl_coin_paprika.py when Airflow triggers the DAG. The call graph shows etl_coin_paprika calls etl_extract_data (outbound HTTP to CoinPaprika) then etl_load_data (writes to PostgreSQL). From there coin_paprika_producer.py takes over: fetch_postgres reads the staged rows and publishes to Kafka. Finally spark_cassandra_etl.py consumes Kafka and writes to Cassandra. That is four hops from API to analytics store, with no internal import edges between the modules — each is a standalone script that communicates only through external services.

The three modules are fully decoupled. etl_coin_paprika.py owns the extract and load functions; coin_paprika_producer.py owns the Kafka publish; spark_cassandra_etl.py owns the transform and Cassandra write. The blast radius is concentrated in etl_coin_paprika.py since both ETL functions route through it, but none of the modules import each other, so changing one does not ripple into the others.

How To Use It

Setup: install dependencies with pip install -e . or uv sync from pyproject.toml. No lockfile is present, so pin versions explicitly if reproducibility matters.

Configuration: the README states the project relies on environment variables for database and service credentials. Load them via a .env file using python-dotenv — the exact variable names are not documented in the repo.

Running it: there is no CLI entry point. The Airflow DAG in etl_coin_paprika.py must be registered with an Airflow instance, then the producer and Spark consumer run as separate processes. Exact commands are not documented; you will need to infer them from the module interfaces.

Real-World Use

A market-monitoring dashboard that needs minute-level price updates. Airflow triggers the DAG every 60 seconds, etl_coin_paprika.py fetches fresh tickers, coin_paprika_producer.py streams them through Kafka, and spark_cassandra_etl.py enriches and stores them. Cassandra serves the dashboard's time-series queries without hammering PostgreSQL.

Code Health & Issues

Static analysis found 3 issues: 1 high, 2 medium.

  • High — Deep nesting in spark_cassandra_etl.py and coin_paprika_producer.py (max indentation depth 8). Control flow is hard to follow; extract guard clauses and inner blocks.
  • Medium — 7 duplicated 6-line code blocks across coin_paprika_producer.py and etl_coin_paprika.py. Extract shared helpers.
  • Medium — No lockfile beside pyproject.toml, so builds are non-reproducible and transitive dependency drift can reach production.
  • Medium — etl_coin_paprika.py makes outbound HTTP calls without a timeout; a stalled peer can hold a worker forever.
  • Med — No test files, no CI/CD pipeline, no LICENSE file. The repo is untested and unlicensed.

The Bottom Line

A clean, decoupled streaming ETL example that demonstrates the full Kafka-to-Spark-to-Cassandra path in about 200 lines of Python. It is not production-ready — no tests, no timeouts, no lockfile — but it is a solid reference for anyone learning streaming data pipelines. Use it as a template, not as a service.