challenges

The Problem

Large, schema‑stable JSON dumps (e.g., Cricsheet match data) occupy gigabytes of storage and strain backup pipelines. Generic compressors leave tens of megabytes on the table because they cannot exploit the predictable field layout.

What This Does

The repo implements a bespoke, schema‑aware compressor that reduced a 2.87 GB cricket dataset to ≈ 42 MB, beating gzip and 7‑zip. Core logic lives in compressor/driver.py (CLI driver) and compressor/main.py (compression primitives). The driver computes the source folder size (folder_size_bytes) and processes each file with compress_one. The actual encoding/decoding algorithms are in compressor/main.py’s compress and decompress functions.

How It Is Wired

Execution starts at the entry point compressor/driver.py line 24:

if __name__ == '__main__':
    main()

main() (driver) performs two distinct actions, reflected in the internal call graph:

1. folder_size_bytes walks the target directory, summing file sizes. It has no outbound calls within the repo, so its blast radius is limited to filesystem reads.
2. compress_one selects a single input file, opens it, and forwards the byte stream to compressor/main.py’s compress function (imported but not listed in the static call graph). compress writes the custom‑encoded output; decompress provides the inverse operation.

Both compress and decompress reside in compressor/main.py. That file contains the only high‑cognitive‑load issue: a nesting depth of eight levels, making the control flow hard to follow. Refactoring with early returns or extracting inner blocks would improve readability and reduce the risk of regression.

No other modules import or are imported by these files, so the import graph is flat (2 modules, 0 edges). Consequently, any change to compressor/main.py propagates only through the two driver calls.

How To Use It

# Clone the repository
git clone https://github.com/moses-y/challenges.git
cd challenges

# Run the compressor on a directory of JSON files
python compressor/driver.py /path/to/cricsheet_json

• The script expects a single positional argument: the root folder containing the JSON files to compress.
• Output files are written alongside the inputs with a custom extension (implementation detail in compress_one).
• To decompress, invoke the same driver with a --decompress flag if the code supports it (check compressor/driver.py for CLI options).

No additional dependencies, Dockerfile, or build steps are present.

Real‑World Use

A data‑engineering pipeline could replace a nightly gzip -9 step with:

import subprocess, pathlib
subprocess.run(['python', 'compressor/driver.py', str(pathlib.Path('/data/cricsheet'))])

The resulting archive is ~20 % smaller than gzip and can be fed directly to downstream analytics jobs that understand the custom format.

Code Health & Issues

• High – Deep nesting – compressor/main.py reaches 8‑level indentation; flattening is recommended.
• Medium – Untested code – No test files detected; core functions lack automated verification.
• Medium – Missing CI – No .github/ workflows or other CI configuration present.
• Medium – No license – Repository lacks a LICENSE file, leaving redistribution rights ambiguous.

No secrets or lockfiles were found.

The Bottom Line

The repository delivers a demonstrably effective, schema‑specific compressor in a minimal Python codebase. Its simplicity makes it easy to integrate, but the lack of tests, CI, and licensing, plus the deep nesting in compressor/main.py, pose maintainability risks. It is suitable for exploratory data‑reduction tasks where the custom format can be safely adopted.