The Problem
Data analysts and modelers need consistent, ready‑to‑use soccer statistics from many public sites. Each source has its own HTML layout, pagination, and identifier scheme, forcing teams to write and maintain separate scrapers. The resulting code is fragile and difficult to integrate into a single analytical pipeline.
What This Does
soccerdata bundles 10 independent scrapers behind a uniform Python API. The core package lives in the soccerdata/ directory; each source has a dedicated module (e.g., fbref.py, espn.py, understat.py). All modules expose a class named after the source that implements methods such as readschedule(), readteamseasonstats(), and readplayerseasonstats().
The package normalises column names and adds common identifiers (team IDs, player IDs) so that a downstream pandas workflow can join data from different sources without custom mapping. Caching is handled automatically (see common.py and config.py), reducing repeated network calls.
How To Use It
Setup
Clone the repo git clone https://github.com/probberechts/soccerdata.git cd soccerdata
Install the library and its runtime deps
pip install . # reads pyproject.toml / requirements.txt optional: install docs dependencies for notebooks pip install -r docs/requirements.txt
Configuration
No global config is required for basic use. Optional JSON files in tests/appdata/config/ (e.g., leaguedict.json) illustrate how a user could map custom league names to the internal identifiers; they are not loaded automatically.
Running it
import soccerdata as sd
Example: FBref data for the 2020/21 English Premier League fb = sd.FBref('ENG-Premier League', '2021') schedule = fb.readschedule() teamstats = fb.readteamseasonstats(stattype='passing') playerstats = fb.readplayerseasonstats(stattype='standard')
All scraper classes follow the same constructor signature (leaguename, season). The returned objects are pandas DataFrames ready for analysis.
Testing
pytest -q
The repository ships 17 test modules under tests/, covering each data source and common utilities.
Real‑World Use
A sports‑analytics team can embed the library in an ETL job:
import pandas as pd, soccerdata as sd
def loadleague(season): fb = sd.FBref('ENG-Premier League', season) df = pd.concat([ fb.readschedule(), fb.readteamseasonstats(stattype='standard'), fb.readplayerseasonstats(stattype='standard') ], axis=1) return df
df2022 = loadleague('2022') df2022.toparquet('premier2022.parquet')
The single loadleague function pulls schedule, team, and player tables in a reproducible format, ready for downstream modeling or dashboarding.
Code Health & Issues
Low – Missing lockfile for docs dependencies – docs/requirements.txt is not version‑pinned; reproducible docs builds may drift. Low – Scraper brittleness – All modules parse live HTML (e.g., fbref.py uses BeautifulSoup). Site layout changes will raise parsing errors; there is limited fallback handling. Low – No explicit rate‑limit / retry logic – Network failures are not uniformly wrapped; a temporary block could crash a batch run. Medium – Tests cover happy paths only – Tests exercise each source once per CI run but do not mock HTTP responses, so CI reliability depends on external sites being reachable. Low – License present – LICENSE.rst declares Apache‑2.0, satisfying compliance. Low – CI/CD configured – GitHub Actions (.github/workflows/ci.yml) run the test suite on push; pre‑commit hooks enforce style (.pre-commit-config.yaml). Low – Documentation completeness – 41 reST files, notebooks, and API reference provide clear usage examples; the README includes a quick‑start snippet.
Overall the repo follows standard Python packaging conventions, includes a lockfile for the main dependencies (uv.lock), and ships a solid test suite.
The Bottom Line
soccerdata delivers a practical, well‑documented Python wrapper for pulling structured soccer statistics from a dozen public sources. It is ready for integration into data pipelines, but users must monitor source‑site changes and consider adding their own retry/back‑off logic for production robustness. Ideal for analysts or small‑to‑medium teams that need quick access to multi‑source soccer data without building scrapers from scratch.