duckle - Moses Yebei

Abstract

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

The Problem

Teams that build ETL pipelines often lock themselves into vendor clouds or pay per‑row fees. They need a way to author pipelines locally, keep the code in source control, and run them on their own infrastructure without vendor lock‑in or hidden costs.

What This Does

duckle is an end‑to‑end ETL/ELT platform that compiles a single pipeline file into SQL that runs inside DuckDB. The engine is written in Rust (see crates/duckdb-engine/), while connectors for CSV, Parquet, PostgreSQL, etc. live in crates/connectors/. A lightweight web console (Dockerfile.web) and a cross‑platform desktop UI (apps/desktop/) provide visual authoring and monitoring. All pipelines are one file in git and can be deployed with duckle-runner serve, which starts the same binary on any machine.

How It Is Wired

Entry points

duckle-runner is built from the workspace in Cargo.toml.
The binary starts in crates/duckle-gpu/src/lib.rs (no main.rs in the crate, the workspace exposes it as a binary).
From there, duckle-gpu calls duckdb-engine::compile_pipeline defined in crates/duckdb-engine/src/lib.rs.
duckdb-engine parses the pipeline definition, loads connectors via connectors::load (crates/connectors/src/lib.rs), and emits a DuckDB SQL string.
The generated SQL is executed against a local DuckDB instance (duckdb_engine::execute_sql).

Side effects

The web console (built with React in frontend/) talks to the runner over a local HTTP endpoint defined in apps/desktop/src/engine_manager.rs.
The desktop UI (apps/desktop/src/main.rs) initializes Tauri, loads the web view, and hooks into the runner’s audit trail stored in a local SQLite database.

Blast radius

duckdb-engine is the core hub; changes here propagate to every pipeline run.
connectors is another hot spot; adding a new data source requires edits in this crate and a re‑build of the runner.
The UI crates are isolated – modifications affect only the user interface.

How To Use It

Setup

git clone https://github.com/moses-y/duckle
cd duckle
cargo build --release

Configuration

Pipeline files live in JSON/YAML (see benchmarks/pg-to-parquet/pipelines/duckle.json.tpl).
Optional secrets are read from apps/desktop/src/secrets.rs; the file is currently empty but can be populated with environment variables (export DB_PASSWORD=…).

Running

./target/release/duckle-runner serve --config path/to/duckle.json

The CLI will start a local DuckDB instance, compile the pipeline, and expose a web console at http://localhost:8000. For the desktop UI: cargo run -p duckle-desktop (or build the Tauri target).

Real‑World Use

A data engineer can author a daily ingest from PostgreSQL to Parquet:

{
  "name": "pg-to-parquet",
  "source": { "type": "postgres", "connection": { "host": "db.local", "user": "app", "password": "${DB_PASSWORD}" } },
  "destination": { "type": "parquet", "path": "/data/output.parquet" },
  "schedule": "0 2 * * *"
}

Place this file in a Git repo, commit, and deploy with duckle-runner serve. The pipeline runs on schedule without any cloud provider.

Code Health & Issues

Low/Security – Secret‑shaped paths discovered in apps/desktop/src/secrets.rs (no secrets committed, but the file exists).
35 test files located under crates/duckdb-engine/src/ and crates/connectors/src/.
109 Markdown and documentation files in docs/ and README.md.
CI defined in .gitlab-ci.yml and several GitHub Actions workflows.
Dual license (MIT & Apache‑2.0) declared in LICENSE-APACHE and LICENSE-MIT.

The Bottom Line

duckle delivers a fully self‑hosted ETL stack that compiles pipelines to native DuckDB SQL, avoiding vendor lock‑in and per‑row costs. It is production‑ready for teams that already use Rust or need a lightweight, open‑source solution. The codebase is well‑structured, with a clear separation between engine, connectors, and UI, but adding new connectors requires touching the core duckdb-engine crate, which can be a maintenance consideration. Overall, it is a solid choice for data engineers who want control over their pipelines and infrastructure.

What the analyser found

Deployment readiness

6/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

730 files

Rust172
JSON103
TSX83
Markdown53
TypeScript50
TOML22

ReactExpressDocker

Module dependencies

The 10 most depended-upon modules of 157, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_frontend_src_tauri_bridge["tauri-bridge"] n1_frontend_src_pipeline_types["pipeline-types"] n2_frontend_src_repo_types["repo-types"] n3_src_workflow_ui_fields_types["types"] n4_kflow_ui_fields_FieldContext["FieldContext"] n5_frontend_src_tauri_dialog["tauri-dialog"] n6_src_workflow_ui_palette_data["palette-data"] n7_frontend_src_tauri_io["tauri-io"] n8_rontend_src_dives_dive_types["dive-types"] n9_i_fields_component_manifests["component-manifests"] n0_frontend_src_tauri_bridge --> n6_src_workflow_ui_palette_data n0_frontend_src_tauri_bridge --> n5_frontend_src_tauri_dialog n0_frontend_src_tauri_bridge --> n1_frontend_src_pipeline_types n7_frontend_src_tauri_io --> n5_frontend_src_tauri_dialog n4_kflow_ui_fields_FieldContext --> n1_frontend_src_pipeline_types n4_kflow_ui_fields_FieldContext --> n2_frontend_src_repo_types n9_i_fields_component_manifests --> n3_src_workflow_ui_fields_types n9_i_fields_component_manifests --> n1_frontend_src_pipeline_types n9_i_fields_component_manifests --> n6_src_workflow_ui_palette_data n9_i_fields_component_manifests --> n0_frontend_src_tauri_bridge n3_src_workflow_ui_fields_types --> n5_frontend_src_tauri_dialog n3_src_workflow_ui_fields_types --> n1_frontend_src_pipeline_types

Dependencies

47 declared dependencies, none a major version behind. Resolved 0 of 47 against the registry.

Findings

LowSecret-shaped paths present; the code-health audit confirms or clears them apps/desktop/src/secrets.rs

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium6

low1

Pin third-party GitHub Actions to a commit SHAhigh6 occurrences

.github/workflows

dtolnay/rust-toolchain@stable, Swatinem/rust-cache@v2, pypa/gh-action-pypi-publish@release/v1, arduino/setup-protoc@v3

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Upgrade the pinned dependency carrying a published advisoryhigh10 occurrences

tar@0.4 CVE-2018-20990 (high) +8 more; tauri@2 CVE-2026-42184 (medium)

The pinned version is the one that installs, so a published advisory against it describes this deployment rather than a hypothetical one, and the advisory is also a public description of how to exploit it.

Fix: Upgrade to the fixed version named in the advisory, then commit the lockfile.

Declare least-privilege permissions for GITHUB_TOKENmedium

.github/workflows/ci.yml

1 workflow(s) declare no permissions, 1 of them reference secrets

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Pin the container base image by digestmedium3 occurrences

Dockerfile.web

node:22-bookworm, rust:1-bookworm, debian:bookworm-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Move large binaries to Git LFS or out of the repositorymedium

website/assets/video/duckle-film.mp4

1 blobs over 5MB: duckle-film.mp4 8.5MB

One repository here carries twenty blobs over 5MB including a 10.9MB spreadsheet, so every clone and every CI checkout pays for data nobody diffs.

Fix: Track those extensions with LFS, or move datasets to object storage and fetch them in a setup step.

Set persist-credentials: false on checkoutmedium

.github/workflows/ci.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add a non-root USER to the imagemedium

Dockerfile.web

CMD or ENTRYPOINT with no USER directive

A process running as root in the container is root against every mounted volume, and it turns any container escape or writable-mount mistake from a contained problem into a host one.

Fix: Create an unprivileged user, chown what it needs, and end the Dockerfile with USER.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/ci.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.