database lab engine

The Problem

PostgreSQL‑based applications need isolated test databases that mirror production state. Creating full copies on every CI run is slow and costly, especially with managed services (RDS, Cloud SQL, Supabase). Teams therefore spend time managing dump/restore scripts or endure flaky tests because migrations run against stale data.

What This Does

DBLab Engine implements thin cloning: it creates a lightweight “branch” of a source cluster using filesystem snapshots (LVM/ZFS) and logical replication. The branch behaves like an independent PostgreSQL instance, letting CI pipelines run full‑stack tests without the overhead of a full restore.

• The core engine lives under engine/. CLI entry points are in engine/cmd/cli/main.go and engine/cmd/database-lab/main.go. Docker images for the server, CLI, and CI checker are defined in the several engine/Dockerfile.* files.
• UI components (React, SCSS) are in ui/ and built with pnpm (ui/package.json). The UI embeds the OpenAPI spec from engine/api/swagger-spec/dblab_openapi.yaml.

Key files:

The observer package (engine/internal/observer/*.go) records clone lifecycle events and feeds metrics to Prometheus (PROMETHEUS.md). The cloning package abstracts the snapshot backend (LVM, ZFS) and provides CreateSnapshot, MountSnapshot, and Cleanup.

The CLI ultimately touches the filesystem (snapshot files), the network (optional SSH tunnel via internal/portfwd/sshtunnel.go), and the PostgreSQL process (via pg_ctl calls inside provision/postgres).

How It Is Wired

1. CLI start – engine/cmd/cli/main.go builds a Cobra root command and registers sub‑commands from engine/cmd/cli/commands/*.
2. User invokes dblab branch create …. Cobra routes to branchAction in engine/cmd/cli/commands/branch/actions.go.
3. branchAction validates flags, loads config (engine/configs/*.yml), then calls cloning.NewBranch.
4. cloning.NewBranch (in engine/internal/cloning/base.go) creates a Branch struct, invokes storage.CreateSnapshot (engine/internal/cloning/storage.go) which selects the backend (LVM via thinclones/lvm/lvm.go or ZFS via thinclones/zfs/branching.go).
5. Snapshot creation triggers observer.NewObserver (engine/internal/observer/observer.go) that registers callbacks for OnCreate, OnMount, OnCleanup. These callbacks publish Prometheus metrics and write audit logs (internal/diagnostic/logs.go).
6. After the snapshot is ready, provision.ProvisionPostgres (engine/internal/provision/databases/postgres/postgres.go) starts a PostgreSQL instance pointing at the mounted snapshot, applying the thin‑clone config files (engine/configs/standard/postgres/...).
7. The CLI returns connection details; downstream CI steps can connect directly to the branch.

Blast radius:

• storage.CreateSnapshot touches block devices – highest risk (requires root).
• observer writes only to local logs/metrics – low risk.
• UI assets are read‑only; they do not affect cloning logic.

No circular dependencies were found; the call graph is a clear tree from CLI → cloning → storage → provision.

How To Use It

# 1. Clone the repo
git clone https://github.com/moses-y/database-lab-engine
cd database-lab-engine

# 2. Build the CLI container (or use local Go)
docker build -f engine/Dockerfile.dblab-cli -t dblab-cli .

# 3. Run the CLI (example creates a branch from a running DB)
docker run --rm -v /var/lib/dblab:/var/lib/dblab \
  dblab-cli branch create \
  --source-uri postgres://user:pass@host:5432/srcdb \
  --branch-name feature‑xyz

Configuration – copy a sample config to engine/configs/config.yaml (e.g., engine/configs/config.example.logical_generic.yml) and adjust snapshot_dir, storage_driver, and any SSH tunnel settings (internal/portfwd/sshtunnel.go).

Running the server – build the server image (engine/Dockerfile.dblab-server) and start it:

docker run -d -p 8080:8080 \
  -v /var/lib/dblab:/var/lib/dblab \
  --name dblab-server \
  dblab-server

The Swagger UI is served at http://localhost:8080/swagger-ui/.

Real‑World Use

In a GitLab CI pipeline, add a job that runs dblab-cli branch create before the test stage, obtains the connection string from the CLI output, runs the test suite against that DB, and finally calls dblab-cli branch delete. This isolates each merge request’s tests without provisioning a full RDS instance.

test:
  stage: test
  image: dblab-cli
  script:
    - export BRANCH=$(dblab-cli branch create --source-uri $SRC_DB_URI --branch-name $CI_COMMIT_SHA)
    - go test ./... -args -db=$BRANCH
    - dblab-cli branch delete $BRANCH

Code Health & Issues

• Low – secret‑shaped path ui/packages/ce/.env (potential credential leak).
• 146 test files exist; coverage appears solid for core logic.
• CI definitions are present (.gitlab-ci.yml, engine/.golangci.yml), but no explicit make test target is documented.
• No license file is missing – LICENSE is present.

No automated static analysis report was supplied, so findings are limited to the directory scan.

The Bottom Line

DBLab Engine delivers a concrete thin‑clone workflow for PostgreSQL, backed by well‑scoped Go packages and a functional CLI. It is production‑ready for CI environments but requires careful handling of snapshot storage privileges and verification of the .env file. Teams comfortable with Docker and Go can adopt it to cut CI DB costs dramatically.