data-formulator Technical Briefing

The Problem

Working with data is hard because data lives everywhere—files, databases, warehouses, BI tools—and connecting AI agents to these sources takes time. As exploration proceeds, questions evolve and a long chat history makes it hard to maintain context. Data Formulator addresses this with one visual workspace for exploring and analyzing data, using data connectors to maintain relationships between sources and data threads to branch into follow-up questions without losing context.

What This Does

Data Formulator is a 689-file AI-powered data analysis system with Python (270 files) and TypeScript/TSX (163 files) codebases. The Python backend lives in py-src/data_formulator/ with 149 files covering data loading, authentication, datalake management, and agent orchestration. The React frontend lives in src/ with 187 files including src/app/App.tsx and src/app/dfSlice.tsx. Key data connectors are defined in py-src/data_formulator/data_connector.py (87 functions, 19 callers) and individual loader modules like athena_data_loader.py, bigquery_data_loader.py, and external_data_loader.py. The system uses Flask on the backend with Docker containerization and yarn for frontend dependencies. Two data files over 5MB (df_gas_prices.json at 5.0MB) are tracked in the repo, and 189 functions make outbound network calls while 38 read/write databases.

How It Is Wired

Execution begins at several entry points. The run function in py-src/data_formulator/agents/agent_chart_restyle.py:104 reaches 399 function(s) and is the primary user-facing start. The execute function in py-src/data_formulator/data_operations/executor.py:46 reaches 222 function(s) and is called from a single place, serving as the data operation worker. Traced paths show execute -> list_tables [db via collection.find().limit, network via boto3.client] and run -> _get_next_action [filesystem via shutil.rmtree]. The get_app_config function in py-src/data_formulator/app.py:372 reaches 53 function(s) and is called from nothing else in the repo, serving as config bootstrap.

The module graph shows src/app/dfSlice as a hub with 46 importers and 14 imports (instability 0.23), participating in a circular dependency cycle alongside src/components/ComponentType, src/app/utils, src/app/apiClient, and src/app/dfSlice. The src/app/layout module has 54 importers but zero outports (instability 0), making it a stable anchor. Eight modules participate in import cycles including dfSlice.tsx, ComponentType.tsx, and utils.tsx — these should be decoupled by extracting shared types or inverting dependencies. The run_app entry point in py-src/data_formulator/app.py:503 reaches 32 function(s) and triggers configure_logging which interfaces with logging.getLogger('openai').setLevel, exposing a model interaction path.

How To Use It

Setup: Install via pip install data_formulator or uvx data_formulator for the latest stable; use pip install --pre data_formulator==0.8.0b1 or uvx data_formulator@0.8.0b1 for beta. The repo provides Dockerfile and docker-compose.yml for containerized deployment.

Configuration: Environment variables and credentials are managed through tests/database-dockers/superset/.env.superset (a committed .env file that must be rotated) and .env.template. The app loads credentials from this file, so any working keys must be rotated if the file is cloned. No .editorconfig or formatter config is present in the repo root.

Running it: Start the backend with py-src/data_formulator/app.py or use the packaged entry point. The frontend runs via npm/yarn commands defined in package.json. For development, the .devcontainer/devcontainer.json provides containerized setup.

Real-World Use

A analyst connects a PostgreSQL database via py-src/data_formulator/data_connector.py, then uses the DataAgent to explore relationships between tables. The agent loads data through data_loader/ modules (e.g., postgresql_data_loader.py), maintains a data memory of source relationships, and branches into follow-up questions using data threads. Visualizations render using Flint chart specs compiled to polished outputs. When the analyst asks for chart restyling, agent_chart_restyle.py:104 executes, reaching 399 functions to adjust chart geometry and theme. Results can be saved as reports through the analyst skills system in py-src/data_formulator/analyst/skills/.

Code Health & Issues

  • HIGH Pin third-party GitHub Actions to commit SHA - .github/workflows uses astral-sh/setup-uv@v7, softprops/action-gh-release@v2, pypa/gh-action-pypi-publish@release/v1 with mutable tags; actions can move and execute with whatever owner last pushed.
  • HIGH Remove committed .env and rotate credentials - tests/database-dockers/superset/.env.superset is tracked and contains live keys; this is the primary leak vector for public clones.
  • MEDIUM Pin container base images by digest - Dockerfile uses node:20-slim, python:3.11-slim without digests; mutable tags mean different libc/CVE sets between builds with no record.
  • MEDIUM Gate pull requests on dependency vulnerability scan - no dependency scan in CI; a known-vulnerable package could reach production without detection.
  • MEDIUM Add pre-commit secret gate - no repo-level gate; a secret-shaped file tracked means the same leak class recurs on every commit.
  • MEDIUM Move large binaries to Git LFS - public/df_gas_prices.json is 5.0MB; every clone and CI checkout pays for undiffed data.
  • LOW Set timeout-minutes on workflow jobs - desktop-build.yml declares no job timeout; a wedged step runs to the six-hour platform default.

The Bottom Line

This repo provides a functional, well-structured AI data analysis system with solid connector coverage and clear entry points, but carries significant operational risk from a committed .env file and mutable GitHub Action tags. The codebase has measurable circular dependencies and cognitive load hotspots in dfSlice.tsx and App.tsx that require refactoring for long-term maintainability. Recommended for teams needing interactive data exploration with AI assistance who can establish credential rotation and dependency pinning processes.