The Problem

Data scientists often need to run analyses on sensitive datasets that remain under the control of a data‑owner organization. Traditional approaches require copying data to a central compute environment, exposing it to compliance risk and adding costly data‑movement pipelines.

What This Does

PySyft implements a privacy‑preserving compute layer that lets a Data Scientist (DS) submit a job definition to a Data Owner (DO) who runs the job inside an isolated virtual environment and returns only approved results. The core runtime lives in the syft/ package (e.g., syft/__init__.py, syft/rds.py in the upstream repo) while concrete service implementations are in the monorepo’s packages:

  • packages/enclave-model-api-example/ – a minimal Flask‑based inference server (src/enclave_model_api/server.py) that demonstrates the enclave API.
  • packages/syft-bg/ – background services for email‑based approval and a TUI dashboard (src/syft_bg/tui/app.py).
  • packages/syft-enclave/ – sandbox execution logic for jobs (src/syft_enclaves/__main__.py).

All client‑side calls import syft as sy and use the high‑level API documented in docs/API.md.

How It Is Wired

  1. Entry point – packages/enclave-model-api-example/src/enclave_model_api/__main__.py runs python -m enclave_model_api. It loads settings.py, creates a Flask app (server.py), and registers routes.
  2. Job submission – the DS calls sy.submit_job(job_spec) (implemented in syft/rds.py). This serialises the job spec to a Google‑Drive‑backed dataset (see docs/connections.md).
  3. Transport – syft/connections/google_drive.py (not listed but present in upstream) writes the job payload to a shared Drive folder. The DO’s enclave daemon (packages/syft-enclave/src/syft_enclaves/__main__.py) watches this folder via watchdog and picks up new jobs.
  4. Approval workflow – packages/syft-bg/src/syft_bg/notify/ contains email templates and Gmail‑watch logic (gmail_watch.py). When a new job appears, syft_bg/api/handlers/job.py triggers an email to the DO. The DO approves via a link that calls back into syft_bg/api/api.py, updating the job state in the Drive dataset.
  5. Isolation – approved jobs are executed by syft_enclaves/exec.py inside a temporary virtualenv created by venv (referenced in packages/syft-enclave/src/syft_enclaves/__init__.py). The job’s stdout/stderr are captured and written back to the Drive results file.
  6. Result retrieval – the DS polls sy.submit_job which reads the result file via the same Drive connector and returns the approved payload.

The most critical hub is the Drive‑based dataset (syft_rds), accessed by both the client SDK and the enclave daemon. Any change to the dataset schema propagates to all three packages, so that module carries the widest blast radius.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/PySyft.git
cd PySyft

# Build the inference‑API container (example)
docker build -t enclave-demo -f packages/enclave-model-api-example/docker/Dockerfile .

# Run the container (the entrypoint starts the Flask server)
docker run -p 5000:5000 enclave-demo

Configuration – OAuth client secrets are read from packages/enclave-model-api-example/src/enclave_model_api/settings.py (GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET). The file expects a credentials.json placed alongside it (not shipped).

Running a job – from a Python REPL on the DS side:

import syft as sy
ds = sy.login_ds(email="ds@example.com")
job = ds.submit_job(
    name="example",
    code="import pandas as pd; df = pd.read_csv('private.csv'); df.describe()"
)
result = job.wait()
print(result)

The client SDK pulls the same pyproject.toml dependency definitions, so pip install . from the repo root also works for local development.

Real‑World Use

A healthcare provider stores patient records in a secured Google Drive. A research team writes a statistical model, packages it as a Python script, and calls sy.submit_job. The provider’s enclave daemon receives the job, runs it in an isolated env with the patient data, and emails the provider for approval. After approval, only aggregate statistics are sent back, satisfying HIPAA constraints while avoiding any data export.

Code Health & Issues

  • Low – Missing lockfile – packages/enclave-model-api-example/docker/requirements.txt lists pip dependencies without a requirements.lock or uv.lock, making container builds non‑reproducible.
  • Medium – Test coverage fragmentation – 188 test files are spread across packages/*/tests/ and the root tests/ directory; no single pytest configuration consolidates them, which can cause missed test discovery in CI.
  • Low – Mixed CI definitions – Multiple GitHub Actions workflows (cd-syft-bg.yml, ci.yml, etc.) each build separate packages; a monorepo‑wide orchestrator would simplify release pipelines.
  • Low – Documentation drift – The README references syft<0.10 for legacy usage, but the repo’s pyproject.toml targets python >=3.10 and publishes syft==0.10+; the version note could confuse new users.

No critical security flaws are evident from static inspection, but the reliance on Google Drive for transport assumes proper OAuth scopes and access controls are enforced externally.

The Bottom Line

PySyft delivers a functional, end‑to‑end privacy‑preserving compute stack built around a Drive‑backed dataset and an email‑driven approval flow. The codebase is modular but spread across many packages, with the enclave daemon and background services being the primary integration points. It is suitable for teams comfortable managing Docker builds and OAuth credentials, but the lack of a lockfile and fragmented CI make reproducible releases harder to guarantee.