openf1 - Moses Yebei

The Problem

Clients that need granular Formula 1 telemetry must stitch together disparate web‑scrapers, live‑timing feeds, and ad‑hoc CSV dumps. Maintaining that pipeline is time‑consuming, error‑prone, and repeats the same parsing logic across projects.

What This Does

openf1 supplies a single Python‑based service that ingests both historic F1 data (scraped from the official site) and live‑timing streams, normalises them, stores the results in MongoDB, and exposes a Flask API for downstream consumption.

Historical scraping lives in src/openf1/services/f1_scraping/ – functions such as download_page (src/openf1/services/f1_scraping/util.py) fetch HTML, while session_result.py and starting_grid.py parse it into Mongo documents.
Live‑timing ingestion resides under src/openf1/services/ingestor_livetiming/. The real‑time entry point is src/openf1/services/ingestor_livetiming/real_time/app.py (Flask app that consumes MQTT topics). A historical back‑fill mode is provided by src/openf1/services/ingestor_livetiming/historical/main.py.
Query access is implemented by src/openf1/services/query_api/app.py, a Flask server that reads from Mongo and returns JSON/CSV.

All code is pure Python; Docker support (Dockerfile, docker‑compose.yml) enables containerised deployment.

How It Is Wired

Execution begins at the Flask entry points:

Real‑time ingest – app.py creates the MQTT client, registers callbacks, then calls process_message. The call chain is: process_message → to_datetime (7×), process_message → _update_lap (5×), process_message → _update_stint (4×). Each update eventually calls upsert_data_sync in src/openf1/util/db.py, which writes to Mongo (collection.bulk_write).
Historical ingest – src/openf1/services/ingestor_livetiming/historical/main.py runs ingest_season. It fetches schedule data via get_schedule (network call in src/openf1/services/f1_scraping/schedule.py), then iterates topics, calling _get_topic_content → requests.get → insert_data_async.
API server – src/openf1/services/query_api/app.py defines endpoint. A request hits _process_request, which parses the path (_parse_path), builds a Mongo query (query_params.py), and finally calls get_documents in src/openf1/util/db.py. The response may also request a favicon via _get_favicon (outbound HTTP).

Core utilities that see the most traffic are src/openf1/util/db.py (15 functions, DB I/O) and src/openf1/util/misc.py (helpers like join_url). The import graph shows no circular dependencies, so changes to a hub file (e.g., objects.py with 16 functions) have a predictable blast radius.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/openf1.git
cd openf1

# Install in editable mode
pip install -e .

# Set Mongo connection (required for all services)
export MONGO_CONNECTION_STRING="mongodb://localhost:27017"

# Run the historical scraper (example for 2023 season)
python -m src.openf1.services.ingestor_livetiming.historical.main ingest_season 2023

# Start the live‑timing ingest
python -m src.openf1.services.ingestor_livetiming.real_time.app

# Launch the query API
python -m src.openf1.services.query_api.app

Docker alternative (build and start all three services) is defined in docker-compose.yml; the Dockerfile builds the Python environment.

Real‑World Use

A data‑science team can schedule ingest_season nightly to back‑fill a MongoDB cache, run the live‑timing ingest continuously during race weekends, and point their analytics notebooks at http://localhost:8000/v1/laps?... to retrieve clean lap‑by‑lap telemetry without writing custom scrapers.

Code Health & Issues

High – GitHub Actions pinned to tags only (JamesIves/github-pages-deploy-action@v4). Replace with commit SHA.
High – No test suite; repository contains 47 source files but zero tests. Add unit/integration tests for each public entry point.
High – No lockfile for Python dependencies (pyproject.toml only). Generate and commit poetry.lock or requirements.txt with hashes.
High – Flask CORS is configured with allow_origins=["*"] together with credentials. Restrict to explicit origins.
Medium – GitHub token permissions are not declared; add permissions: block.
Medium – Dependabot/Renovate not configured; add .github/dependabot.yml.
Medium – Docker base image python:3.10-slim is unpinned; use digest.
Medium – No CI step for dependency vulnerability scanning; add dependency-review-action or osv-scanner.
Medium – Outbound requests.get calls lack timeouts; pass timeout=.
Medium – actions/checkout keeps credentials; set persist-credentials: false.

Additional observations: the repository includes a CI workflow (.github/workflows/tests.yml) but no test files, and the license file is present. No secrets are committed.

The Bottom Line

openf1 delivers a functional end‑to‑end pipeline for F1 data ingestion and API delivery, with clear separation between scraping, live‑timing, and query layers. However, the lack of tests, an unlocked dependency set, and several security‑related CI misconfigurations make it unsuitable for production without remediation. It is a solid starting point for internal tooling or research prototypes, provided the team adds test coverage and tightens the CI/CD hygiene.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

90 files

Python36
Markdown15
JavaScript11
SCSS6
YAML4
HTML1

FlaskDocker

Findings

LowDependencies declared without a lockfile - non-reproducible builds pyproject.toml

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high4

medium7

low1

Pin third-party GitHub Actions to a commit SHAhigh

.github/workflows

JamesIves/github-pages-deploy-action@v4

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Add a test suite; this repository has nonehigh47 occurrences

47 source files, no test files

Any change ships with no signal that existing behaviour still holds, so a regression reaches production undetected.

Fix: Add one test per public entry point, then a CI step that runs them.

Commit a lockfile beside the manifesthigh

pyproject.toml

manifest with no lockfile in scope

An unlocked range means the artifact you tested and the artifact you ship can contain different transitive code, so a malicious patch release reaches production with no diff.

Fix: Run the package manager once and commit the generated lockfile.

Replace the wildcard CORS origin with an explicit allow listhigh

src/openf1/services/query_api/app.py

allow_origins=["*"] together with credentials enabled

Any page on the internet can call the API with the browser's cookies attached, so a logged-in visitor to an unrelated site performs authenticated requests without knowing, and the wildcard combined with credentials is the exact configuration browsers refuse for that reason.

Fix: List the origins the API actually serves, and never pair a wildcard with credentials.

Declare least-privilege permissions for GITHUB_TOKENmedium2 occurrences

.github/workflows/tests.yml

2 workflow(s) declare no permissions, 1 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Enable Dependabot or Renovatemedium

2 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium

Dockerfile

python:3.10-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Give the outbound request a timeoutmedium2 occurrences

src/openf1/services/ingestor_livetiming/historical/main.py

2 outbound call(s) with no timeout

requests has no default timeout, so a peer that accepts the connection and then says nothing holds the worker forever, and with a small worker pool a single slow dependency takes the whole service down without anything appearing to fail.

Fix: Pass timeout= on every call, or use a session with one configured.

Set persist-credentials: false on checkoutmedium

.github/workflows/tests.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

Dockerfile

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Set timeout-minutes on the workflow jobslow2 occurrences

.github/workflows/tests.yml

2 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.