The Problem

AI agents that rely on retrieval‑augmented generation need constantly‑fresh context. Traditional batch pipelines re‑process whole corpora, creating latency and “stale‑data” gaps that hurt answer relevance and increase compute cost.

What This Does

cocoindex provides an incremental sync engine that watches arbitrary data sources (code, Slack, PDFs, video captions, etc.) and re‑indexes only the delta. The core Python package lives under python/cocoindex/_internal/ and implements the declarative pipeline, state tracking, and target‑store adapters. Rust components in rust/ supply high‑performance vector‑store back‑ends and transformation utilities. Example applications in examples/ (e.g., examples/audio_to_text/main.py and examples/amazon_s3_embedding/main.py) show how to wire a source connector to a target such as a Neo4j graph or a vector DB.

How It Is Wired

Execution begins at a concrete script, for instance examples/audio_to_text/main.py. That file:

  1. Loads configuration from the adjacent .env (currently committed – see health section).
  2. Instantiates a source connector (AudioSource in examples/audio_to_text/connector.py) and a target sink (EmbeddingStore in python/cocoindex/_internal/target_state.py).
  3. Calls cocoindex._internal.api.run_pipeline(source, target) – the central function in python/cocoindex/_internal/api.py.

api.run_pipeline orchestrates the flow:

  • State handling – via python/cocoindex/_internal/state.py (creates/updates checkpoint files).
  • Component execution – each pipeline step is a live component (python/cocoindex/_internal/live_component.py) that registers a coroutine in the component context (python/cocoindex/_internal/component_ctx.py).
  • Function wrappers – the heavy‑lifting logic resides in python/cocoindex/_internal/function.py, which contains 1,835 lines of code and is the widest blast‑radius module (high cognitive load).

The call graph shows that api.py imports 18 other internal modules while being imported by only two, making it a hub with high instability (0.9) and part of a circular import cycle that also involves __init__.py, component_ctx.py, and context_keys.py. Those cycles increase the risk that a change in one module propagates unexpectedly to many others.

Rust side: benchmarks such as benchmarks/file_summarization/rust/src/main.rs compile the cocoindex core (rust/Cargo.toml) into a binary that calls the same internal API via FFI. The Rust crate provides the vector‑store implementation (rust/src/vector_store.rs) and is invoked directly from the Python wrapper when a target requires high‑throughput similarity search.

Overall, a typical run follows ≈4 hops: entry script → API → live component → function → external store (e.g., Neo4j, S3, or a local vector DB). Modules like target_state.py and live_component.py own the side‑effects (network I/O, file writes); they should be the focus of any change.

How To Use It

# Clone the repository (use the exact URL)
git clone https://github.com/moses-y/cocoindex
cd cocoindex

# Build Rust components
cargo build --release   # uses Cargo.toml at repo root

# Install Python parts (example with uv, fallback to pip)
uv pip install -e python/   # editable install, resolves pyproject.toml
# or:
pip install -e python/

# Create a local .env (do NOT commit)
cp .env.lib_debug .env.example
# edit .env to add your source keys (e.g., AWS, Azure) and target creds

# Run an example – audio to text embedding
python examples/audio_to_text/main.py

The repository already contains a Dockerfile (root) for containerised execution; building it will install both Rust and Python layers.

Real‑World Use

A SaaS product that ingests customer support tickets nightly can embed this engine as follows:

from cocoindex._internal.api import run_pipeline
from my_connectors import TicketSource
from my_targets import PineconeSink

pipeline = run_pipeline(TicketSource(), PineconeSink())
pipeline.start()   # processes only new tickets since last run

Only newly‑arrived tickets trigger embedding and upsert operations, keeping the vector index fresh without re‑indexing the entire backlog.

Code Health & Issues

  • High – GitHub Actions pinned to tags only (.github/workflows/*). Replace with commit SHAs.
  • High – Committed .env files (.env.lib_debug, examples/*.env). Remove, add to .gitignore, rotate credentials.
  • High – CI does not run the test suite despite 170 test files. Add a pytest step to the workflows.
  • High – Docs release workflow pushes directly to gh-pages. Switch to a PR‑based deployment.
  • Medium – No dependency‑vulnerability scan in CI. Add dependency-review-action or osv-scanner.
  • Medium – Checkout step keeps credentials; set persist-credentials: false in release.yml.
  • Low – Workflow jobs lack explicit timeout-minutes. Define reasonable limits.

Measured static analysis also found:

  • 5 import cycles involving core internal modules (__init__.py, component_ctx.py, context_keys.py). Break cycles to improve stability.
  • 46 deep‑nesting spots (max indentation 8) in files like function.py and environment.py. Refactor for readability.
  • Broad exception handling in several internal modules; narrow catches to avoid silent failures.
  • Oversized files (function.py – 1,835 lines). Split into focused sub‑modules.

No missing license or test framework; a license file is present.

The Bottom Line

cocoindex delivers a functional incremental indexing framework with both Python and Rust components, suitable for production AI agents that need near‑real‑time context. The codebase is modular but suffers from import cycles, overly large core files, and several CI/secret‑management hygiene issues that must be addressed before safe deployment in a security‑sensitive environment. Teams comfortable with Python‑centric pipelines and willing to clean up the health concerns will find it a valuable foundation.