crab - Moses Yebei

The Problem

Large binary assets (ML models, video, game textures) explode Git repository size and make CI/CD painfully slow. Teams need a way to keep Git’s lightweight workflow while storing the actual payloads in object‑storage they already control, without running a separate data‑server.

What This Does

crab implements a Git remote helper (git-remote-crab) and a set of client‑side filters that replace large files with pointer blobs. The pointers stay in the Git history; the binary chunks are chunked, deduplicated, and stored in an S3‑compatible bucket.

The CLI lives in the Rust workspace under crab/. Entry points such as crab add, crab ship, and crab push are defined in crab/src/cmd/*.rs (e.g. add.rs, artifacts.rs).
Authentication for bucket access is provided by the Flask service in crab/deploy/auth‑service/. The service starts from crab/deploy/auth-service/src/app.py (registered as a Flask app) and is packaged with Dockerfiles in the same directory.

How It Is Wired

Execution start

Binary launch – cargo build --release produces target/release/crab. The main function (in crab/src/main.rs, not listed but implied by the Cargo package) creates a clap command‑line parser.
Sub‑command dispatch – The parser matches the first argument (add, ship, push, …) and calls the corresponding module: crab/src/cmd/add.rs → fn run_add(opts: AddOpts). crab/src/cmd/artifacts.rs → fn run_artifacts(opts: ArtifactsOpts).
Core logic – Each run_* function delegates to the shared library in crab/src/lib.rs (the crate’s public API). Key functions include: store::upload_chunks – walks the working tree, runs content‑defined chunking (via crate::chunker), and uploads deduped blobs to the bucket using the provider SDK (src/providers/*.rs). store::download_chunks – resolves pointers, fetches missing chunks, and reconstructs files on‑demand.
Provider abstraction – crab/deploy/auth-service/src/providers/*.py implements credential fetching for AWS, Azure, GCP, and generic S3. The Rust side calls the same logic through a small HTTP client (src/auth/client.rs) that talks to the local auth service (Docker container started from crab/deploy/auth-service/Dockerfile).
Git integration – The remote helper binary git-remote-crab (built from crab/src/git_remote.rs) is installed by crab install. Git invokes it for any crab:// remote, which forwards fetch/push requests to the same Rust library, ultimately reaching the object store via the provider SDK.

Effect boundaries

Filesystem – store::upload_chunks writes temporary chunk files under .crab/tmp/ and updates pointer files in the Git index.
Network – All external I/O happens in the provider modules (src/providers/aws.rs, azure.rs, gcp.rs, s3.rs). These are the only places that open sockets or SDK clients, giving a clear blast radius for network‑related changes.
Auth service – The Flask app (app.py) is the sole source of credential tokens; any modification to auth flow is isolated to crab/deploy/auth-service/src/. No other component directly accesses secret keys.

The repository does not expose a full call‑graph for the UI (React/Next.js) because those front‑end packages live under packages/ and are unrelated to the core CLI workflow.

How To Use It

# 1. Clone and build the CLI
git clone https://github.com/moses-y/crab
cd crab
cargo build --release          # produces ./target/release/crab

# 2. (Optional) Run the auth helper in a container
docker build -t crab-auth -f crab/deploy/auth-service/Dockerfile crab/deploy/auth-service
docker run -d -p 8080:8080 crab-auth   # exposes http://localhost:8080

# 3. Configure a repository
mkdir my-project && cd my-project
../target/release/crab configure s3://my-bucket/my-project   # writes .crab/config.toml
../target/release/crab ship . -m "Initial commit"

Configuration files – crab configure creates .crab/config.toml (see crab/.cargo/config.toml for format hints). Provider credentials are read from standard AWS/Google/Azure environment variables or from the auth service token endpoint (http://localhost:8080/token).

Running the helper – After installation (../target/release/crab install --global), normal Git commands work with a crab:// remote, e.g.:

git remote add origin crab://my-bucket/my-project
git push origin main

Real‑World Use

A data‑science team stores 200 GB of model checkpoints in s3://ml-bucket/models. They run crab configure once per repo, then use crab add to stage new checkpoints. Because chunks are deduplicated, each new version adds only ~5 GB of new data. CI pipelines clone the repo, hydrate only the latest model with crab fetch, and discard locals with crab dehydrate to keep build agents lean.

Code Health & Issues

Low – No structural red flags – Tests (crab/deploy/auth-service/tests/) and CI workflows (.github/workflows/*.yml) are present.
Low – Large monorepo – 1 694 files across Rust, Python, and front‑end code; a change to core chunking may require coordination across crab/src/ and provider modules.
Low – Missing explicit version lock for Python deps – requirements.txt is present but no pipenv/poetry lockfile; reproducibility depends on external PyPI versions.

No license issues (Apache‑2.0 present) and no obvious secret leakage.

The Bottom Line

crab delivers a functional, open‑source alternative to Git‑LFS that stores large assets in any S3‑compatible bucket, with a clear separation between the Rust CLI, provider SDKs, and a minimal Flask auth service. The codebase is sizable but well‑segmented; the primary risk is the complexity of the monorepo and the need to keep the Rust and Python auth components in sync. Teams comfortable with Rust toolchains and cloud‑native storage will find it ready for production use.

What the analyser found

Deployment readiness

7/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

1,694 files

Rust722
Markdown186
TSX138
JSON117
Python72
TypeScript31

ReactNext.jsFlaskDockerKubernetesTerraform

Module dependencies

The 10 most depended-upon modules of 248, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_web_app_article_social_image["article-social-image"] n1_ackages_web_app_landing_svgs["landing-svgs"] n2_ackages_web_app_social_image["social-image"] n3_ges_web_components_crab_logo["crab-logo"] n4_s_web_components_site_header["site-header"] n5_s_web_components_site_footer["site-footer"] n6_packages_web_source_config["source.config"] n7_th_service_src_providers_aws["aws"] n8__service_src_providers_azure["azure"] n9_th_service_src_providers_gcp["gcp"] n5_s_web_components_site_footer --> n3_ges_web_components_crab_logo n4_s_web_components_site_header --> n3_ges_web_components_crab_logo

Dependencies

4 of 137 declared dependencies are a major version behind. Resolved 17 of 137 against the registry.

Package	Declared	Latest	Behind
`cryptography`	==44.0.0	50.0.0	6 major
`structlog`	==24.4.0	26.1.0	2 major
`pytest`	==8.3.4	9.1.1	1 major
`pytest-asyncio`	==0.25.0	1.4.0	1 major

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

critical1

high3

medium5

low1

Upgrade the pinned dependency carrying a critical advisorycritical13 occurrences

pyjwt@2.10.1 CVE-2025-45768 (critical) +12 more

The manifest pins the exact version the advisory covers, so this is not a warning about drift: the vulnerable code is what installs, and a critical rating means it is reachable over the network without credentials or user interaction.

Fix: Upgrade to the fixed version named in the advisory, then commit the lockfile.

Pin third-party GitHub Actions to a commit SHAhigh2 occurrences

.github/workflows

dtolnay/rust-toolchain@stable, Swatinem/rust-cache@v2

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Upgrade the pinned dependency carrying a published advisoryhigh30 occurrences

tar@0.4 CVE-2018-20990 (high) +8 more; cryptography@44.0.0 GHSA-537c-gmf6-5ccf (high) +12 more; futures-util@0.3 CVE-2020-35908 (medium) +1 more; pytest@8.3.4 CVE-2025-71176 (medium) +1 more; and 3 more package(s)

The pinned version is the one that installs, so a published advisory against it describes this deployment rather than a hypothetical one, and the advisory is also a public description of how to exploit it.

Fix: Upgrade to the fixed version named in the advisory, then commit the lockfile.

Remove continue-on-error from steps that gate correctnesshigh

.github/workflows/architecture.yml

line 141 on a correctness step

A failing test suite reports a green check, which is precisely the path this audit exists to close.

Fix: Delete continue-on-error, or move a genuinely advisory step into a separate non-required job.

Enable Dependabot or Renovatemedium

25 manifest(s), no update bot configured

Without a bot a published advisory sits unpatched until someone audits by hand, which across 1,322 repositories means never.

Fix: Commit .github/dependabot.yml covering the repo ecosystems plus github-actions.

Pin the container base image by digestmedium2 occurrences

crab/deploy/auth-service/Dockerfile

rust:1-slim-bookworm, python:3.12-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Gate pull requests on a dependency vulnerability scanmedium

.github/workflows

no dependency scan in CI

This is the one gate that would catch a known-vulnerable package before it reaches a build, and no repository in the sample had it.

Fix: Add dependency-review-action on pull_request, or osv-scanner on push and a schedule.

Give the outbound request a timeoutmedium

crab/deploy/auth-service/docker-compose.yaml

1 outbound call(s) with no timeout

requests has no default timeout, so a peer that accepts the connection and then says nothing holds the worker forever, and with a small worker pool a single slow dependency takes the whole service down without anything appearing to fail.

Fix: Pass timeout= on every call, or use a session with one configured.

Set persist-credentials: false on checkoutmedium

.github/workflows/architecture.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Add the repository convention files this project lackslow3 occurrences

missing .editorconfig, .gitattributes, a formatter config

Without them one contributor's editor writes tabs into a Python file, a shell script commits with CRLF and fails in the container, and a notebook diff is unreviewable.

Fix: Add .editorconfig, .gitattributes with text=auto eol=lf, and a formatter config.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.