The Problem Enterprises need to answer questions from private documents without exposing the data to external services. Existing “search‑then‑LLM” pipelines are often hard‑coded to a single vector store or embedding model, making it difficult to swap components or comply with security policies.

What This Does deep-searcher provides a modular RAG stack that can ingest local files or crawl web pages, embed them with a selectable provider, store vectors in a configurable DB (Milvus, Qdrant, Azure Search, Oracle, etc.), and answer queries using a wide range of LLM back‑ends (OpenAI, Anthropic, Gemini, Ollama, Bedrock, etc.).

Key files:

  • deepsearcher/cli.py – command‑line entry point (deepsearcher command).
  • deepsearcher/online_query.py – high‑level query() function used by the CLI and examples.
  • deepsearcher/offline_loading.py – loaders for local files (load_from_local_files) and web crawlers (load_from_website).
  • deepsearcher/vector_db/*.py – adapters that implement a common BaseVectorDB interface.
  • deepsearcher/embedding/*.py – concrete embedding classes each exposing a embed_documents() method.
  • deepsearcher/llm/*.py – LLM wrappers exposing a complete() method.

The package ships with a deepsearcher/config.yaml default config and a programmatic deepsearcher/configuration.py API for runtime overrides.

How It Is Wired

  1. Startup – deepsearcher/cli.py parses arguments and loads the global Configuration (deepsearcher/configuration.py).
  2. Provider init – Configuration.set_provider_config() selects concrete classes via the provider_registry maps in embedding/__init__.py, llm/__init__.py, and vector_db/__init__.py. Each chosen class is instantiated once and stored in the config singleton.
  3. Data ingestion – offline_loading.load_from_local_files() walks supplied paths, delegates to file‑type loaders under loader/file_loader/ (PDF, text, JSON, etc.), then calls the selected embedding class to produce vectors and finally vector_db.insert() to persist them. Web crawling follows the same path via loader/web_crawler/.
  4. Query – online_query.query() receives a user prompt, retrieves relevant chunks with vector_db.search(), builds a prompt, calls the LLM’s complete(), and returns the LLM response plus optional source citations.
  5. Exit – CLI prints the result and exits; main.py simply forwards to cli.main() for programmatic use.

The widest blast radius lies in the Configuration singleton: any change to provider names or model parameters propagates to all downstream components. The vector‑DB adapters are the primary I/O boundary (network, disk), while the embedding and LLM wrappers are thin HTTP clients.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/deep-searcher && cd deep-searcher

# Install (recommended uv for reproducibility)
uv sync          # reads pyproject.toml, creates .venv
source .venv/bin/activate

# Optional: install extra LLM/embedding back‑ends
pip install "deepsearcher[ollama]"   # example extra

# Load data (example)
python - <<'PY'
from deepsearcher.configuration import Configuration, init_config
from deepsearcher.offline_loading import load_from_local_files
cfg = Configuration()
cfg.set_provider_config("embedding", "OpenAIEmbedding",
                        {"model": "text-embedding-ada-002"})
cfg.set_provider_config("vector_db", "Milvus", {"host": "localhost", "port": 19530})
init_config(config=cfg)
load_from_local_files(paths_or_directory="examples/data")
PY

# Run a query via CLI
export OPENAI_API_KEY=sk-...
deepsearcher query "Write a brief report about Milvus."

Real‑World Use A compliance team can point load_from_local_files() at a directory of policy PDFs, choose an on‑prem Milvus instance for vector storage, and configure OpenAI or an internal LLM for answering. The same pipeline can later be switched to Azure Search and Anthropic by editing config.yaml—no code changes required.

Code Health & Issues

  • Low – No lockfile – Dependencies are listed only in pyproject.toml; reproducible builds rely on external lock generation.
  • Medium – CI coverage – GitHub Actions run lint (ruff.yml) and docs builds but no explicit test execution workflow; tests exist but are not automatically validated.
  • Low – Documentation gaps – docs/usage/cli.md and docs/quick_start.md cover basic flow, but required environment variables for each LLM are scattered across provider modules, not centralized.
  • Low – Dockerfile – Provides a container image but lacks multi‑stage build optimizations; the image size is not constrained.

The Bottom Line deep-searcher delivers a well‑structured, extensible RAG framework with clear separation of loaders, embeddings, vector stores, and LLMs. It is suitable for teams that need to swap back‑ends without touching core logic. The main drawbacks are the absence of a lockfile and incomplete CI test automation, which could affect reproducibility and release confidence. Engineers comfortable with Python packaging and containerisation will find the codebase approachable and ready for production integration.