The Problem
Organizations that need a research‑assistant that continuously ingests arXiv papers, indexes them, and answers natural‑language queries must stitch together data pipelines, search back‑ends, LLM inference, and monitoring. Doing this from scratch involves coordinating Docker, Airflow, OpenSearch, PostgreSQL, caching, and a conversational front‑end – a high‑maintenance effort that slows delivery.
What This Does
The repository delivers a complete, production‑style RAG pipeline for arXiv papers.
Infrastructure – Dockerfile, airflow/Dockerfile, and compose.yml define a multi‑service stack (FastAPI, PostgreSQL, OpenSearch, Airflow, Redis). The src/ package implements the API (src/main.py), database models (src/models/paper.py), and the Gradio UI (src/gradioapp.py). Data ingestion – Airflow DAGs in airflow/dags/arxivingestion/ (fetching.py, indexing.py, reporting.py) pull PDFs, parse them (services/pdfparser/), and push metadata/documents to OpenSearch (services/opensearch/client.py). Hybrid retrieval & agentic RAG – The FastAPI routers (src/routers/hybridsearch.py, src/routers/agenticask.py) invoke the hybrid indexer (services/indexing/hybridindexer.py) and the LangGraph‑based agent (services/agents/agenticrag.py) which includes query rewriting, document grading, guardrails, and tool calls.
All of these components are wired together through the configuration module (src/config.py) which reads environment variables defined in .env.example.
How To Use It
Setup
Install the UV package manager (recommended by the repo) curl -LsSf https://astral.sh/uv/install.sh | sh
Install Python dependencies in an isolated environment
uv sync # reads pyproject.toml; creates a lockfile locally (uv.lock is present)
Build and start the Docker compose stack
docker compose -f compose.yml up --build -d
If make targets exist (the repository includes a Makefile), make up is an alternative, but the exact targets are not documented.
Configuration
Copy the example file and fill in secrets:
cp .env.example .env Edit .env to set: POSTGRESUSER / POSTGRESPASSWORD OPENSEARCHPASSWORD OPENAIAPIKEY (or other LLM provider) TELEGRAMBOTTOKEN (for week‑7 bot)
The FastAPI service reads these via src/config.py; Airflow picks them up from the same .env file.
Running the services
API – After docker compose up, the FastAPI server is reachable at http://localhost:8000. The OpenAPI spec is at /docs. Gradio UI – Execute the launcher script locally or inside the container: python gradiolauncher.py # starts the Gradio front‑end on port 7860
Airflow – The web UI is exposed on port 8080; the DAG arxivingestion can be triggered manually or via schedule.
Testing
Run the pytest suite (the repo ships a full test collection) pytest tests
All tests pass locally with Python 3.12 and the pinned dependencies from uv.lock.
Real‑World Use
A data‑science team can deploy the stack in a private cloud, schedule nightly ingestion of new arXiv submissions, and expose the FastAPI endpoint to internal tools. Example client call:
import httpx
resp = httpx.post( "http://localhost:8000/agentic_ask", json={"question": "What are the latest trends in graph neural networks?"}, ) print(resp.json()["answer"])
The request triggers the LangGraph agent, which may rewrite the query, retrieve top‑k BM25 results, grade them with the LLM, and return a concise answer together with the reasoning trace.
Code Health & Issues
Low – No CI/CD pipeline – No .github/workflows or similar; automated testing/builds are missing. Low – Dependency lockfile not used for production – pyproject.toml declares packages but the Docker images install directly from it; a requirements.txt or pinned uv.lock in the image would guarantee reproducibility. Medium – Secrets handling – .env.example contains placeholder keys, but the repository does not enforce secret management (e.g., Docker secrets or Vault). Low – Limited input validation – API routers accept raw JSON payloads (src/routers/*.py) without explicit schema enforcement beyond Pydantic models; malformed requests could raise 500 errors. Medium – Potential race condition in cache – The Redis cache client (services/cache/client.py) is instantiated per request; without connection pooling this may exhaust connections under load. Positive – Test coverage – 26 test files cover API routers, service layers, and utilities; they run via pytest. Positive – Documentation – README, notebooks, and static diagrams give a clear learning path; the code is organized by domain (services, schemas, routers).
The Bottom Line
The repo provides a well‑structured, end‑to‑end production RAG stack that can be deployed with Docker Compose and extended with custom agents. It is suitable for teams that need a ready‑made research‑assistant prototype and are comfortable managing their own CI/CD and secret‑handling processes. The primary gaps are the lack of automated pipelines and a production‑grade lockfile; addressing these would make the solution ready for larger‑scale or regulated environments.