pandaprobe - Moses Yebei

Abstract

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

pandaprobe Technical Briefing

Clone: git clone https://github.com/moses-y/pandaprobe.git

The Problem AI agent deployments lack centralized observability. Teams building with frameworks like LangGraph or CrewAI face fragmented tracing and no standardized evaluation pipeline, making it difficult to debug failures or measure agent reliability across iterations.

What This Does PandaProbe is a full-stack platform comprising a Python backend (173 files) and a TypeScript/React frontend (149 files). It provides tracing infrastructure and a metric library for agent evaluation. The backend is architected around a FastAPI-style router (backend/app/api/v1/router.py) that dispatches to route handlers in backend/app/api/v1/routes/, including traces.py, evaluations.py, and sessions.py. Tracing data flows into an ORM layer defined in backend/app/infrastructure/db/models.py and is accessed via repositories in backend/app/infrastructure/db/repositories/. Evaluation metrics are implemented as modular Python classes under backend/app/core/evals/metrics/, covering trace coherence, step efficiency, and tool correctness, with schemas defined in sibling schema.py files. The LLM integration layer resides in backend/app/infrastructure/llm/, abstracting provider calls through engine.py and providers.py.

How It Is Wired

Entry Points: API requests route through backend/app/api/v1/routes/cli.py (CLI) or the main router. The frontend API client is at frontend/src/lib/api/cli.ts.
Call Graph: Ingestion $\rightarrow$ traces.py route $\rightarrow$ trace_repo (DB write) $\rightarrow$ trace_service $\rightarrow$ Dashboard display. Evaluation runs are triggered via eval_service (backend/app/services/eval_service.py), which iterates over metric definitions to compute scores stored in the eval_runs table (migrations reflect this evolution, e.g., 572b122d002a_refactor_evaluations_to_eval_runs_and_).
External Dependencies: The system integrates with Redis (caching/queues via backend/app/infrastructure/redis/) and Celery (backend/app/infrastructure/queue/celery_app.py) for background task processing. LLM providers are configured via backend/app/infrastructure/llm/providers.py.

How To Use It

Setup: Clone the repository. The project uses yarn as the package manager (frontend) and standard Python tooling (backend). Docker is required for self-hosting.
Configuration: Environment variables are defined in backend/.env.example. Key settings typically include database connection strings and LLM API keys.
Running it: The provided ./start.sh script orchestrates the stack via docker-compose.yml. Once running, the Dashboard is available at http://localhost:3000 and the API reference at http://localhost:8000/scalar.

Real-World Use A team iterating on a LangGraph agent can use the SDK integration (referenced in docs) to automatically emit traces to the local PandaProbe instance. As the agent runs, spans appear in the dashboard. The team can then define a custom metric—extending the patterns in backend/app/core/evals/metrics/trace/—to evaluate plan adherence or tool correctness on a dataset of historical runs, visualizing the trade-offs directly in the UI.

Code Health & Issues Static analysis of the structure indicates a mature SDLC: 488 files, 59 test files (unit and integration), and a complete CI/CD pipeline (GitHub Actions: build, lint, test-unit, test-integration, codeql). The license is Apache 2.0. No structural red flags were detected. The codebase is evenly split between Python and TypeScript/React, with a clear separation of concerns between infrastructure, services, and API routing.

The Bottom Line PandaProbe offers a functional, self-hostable stack for agent tracing and evaluation without the immediate complexity of a SaaS commitment. The code is well-organized around a router/service/repository pattern, and the metric framework is extensible. It is well-suited for teams already using LangGraph or CrewAI who need to add observability and eval loops to their existing pipelines. The primary trade-off is the operational overhead of self-hosting versus the cloud offering, though the Docker setup mitigates this significantly.

What the analyser found

Deployment readiness

7/7

Container image
CI pipeline
Lockfile committed
Test suite
README
License
No committed secrets

Composition

488 files

Python173
TSX99
TypeScript50
YAML13
JSON12
Markdown9

ReactNext.jsDjangoFlaskDockerLangChain

Module dependencies

The 10 most depended-upon modules of 309, from static import analysis. Red outline marks a module in an import cycle.

graph LR n0_frontend_src_lib_api_client["client"] n1__tests_integration_factories["factories"] n2_frontend_src_lib_api_types["types"] n3_nents_providers_AuthProvider["AuthProvider"] n4_frontend_src_lib_api_enums["enums"] n5_d_tests_integration_conftest["conftest"] n6_s_features_InstructionShared["InstructionShared"] n7_ts_providers_PostHogProvider["PostHogProvider"] n8_frontend_src_lib_auth_config["config"] n9__features_InstructionContent["InstructionContent"] n5_d_tests_integration_conftest --> n1__tests_integration_factories n7_ts_providers_PostHogProvider --> n3_nents_providers_AuthProvider n2_frontend_src_lib_api_types --> n4_frontend_src_lib_api_enums

Dependencies

8 of 76 declared dependencies are a major version behind. Resolved 76 of 76 against the registry.

Package	Declared	Latest	Behind
`@types/node`	^20	26.2.0	6 major
`redis`	>=5.2.0	8.1.0	3 major
`structlog`	>=24.4.0	26.1.0	2 major
`typescript`	^5	7.0.2	2 major
`firebase-admin`	>=6.6.0	7.5.0	1 major
`react-day-picker`	^9.14.0	10.0.1	1 major
`@testing-library/jest-dom`	^6.9.1	7.0.1	1 major
`eslint`	^9	10.8.1	1 major

What would stop me shipping this

Ranked by severity × confidence × production reach. Reach is the honest discriminator across a collection that is mostly other people's code: the same finding matters more in something that ships.

high2

medium3

low2

Pin third-party GitHub Actions to a commit SHAhigh8 occurrences

.github/workflows

astral-sh/setup-uv@v7, docker/setup-buildx-action@v4, docker/login-action@v4, docker/metadata-action@v6

A tag can be moved, so the action running with your token and secrets is whatever its owner last pushed; this is how tj-actions/changed-files leaked secrets from thousands of repos.

Fix: Replace each @vN with the 40-character commit SHA, keep # vN as a comment, and let Dependabot bump the SHAs.

Upgrade the pinned dependency carrying a published advisoryhigh22 occurrences

next@16.2.3 CVE-2026-44575 (high) +21 more

The pinned version is the one that installs, so a published advisory against it describes this deployment rather than a hypothetical one, and the advisory is also a public description of how to exploit it.

Fix: Upgrade to the fixed version named in the advisory, then commit the lockfile.

Declare least-privilege permissions for GITHUB_TOKENmedium3 occurrences

.github/workflows/build.yml

3 workflow(s) declare no permissions

With no declaration the token inherits the repository default, so any injected step can push commits or mint releases from inside your own CI.

Fix: Add permissions: contents: read at the top of the workflow and widen per job only where needed.

Pin the container base image by digestmedium

backend/Dockerfile

python:3.13.2-slim

An untagged or mutable base means today's build and last month's contain different libc and a different CVE set, with no record of which shipped.

Fix: Use image:tag@sha256:<digest> and enable Dependabot's docker ecosystem.

Set persist-credentials: false on checkoutmedium

.github/workflows/build.yml

checkout keeps the token, then dependencies are installed

The token stays in .git/config for every later step, so a malicious postinstall script reads a pushable credential without one ever being passed to it.

Fix: Add with: persist-credentials: false, and pass an explicit token only to the step that pushes.

Set timeout-minutes on the workflow jobslow4 occurrences

.github/workflows/build.yml

4 workflow(s) declare no job timeout

A wedged step runs to the six-hour platform default, which on a two-hourly schedule means three runs overlap behind it.

Fix: Add timeout-minutes with a realistic bound to each job.

Add the repository convention files this project lackslow3 occurrences

missing .editorconfig, .gitattributes, a formatter config

Without them one contributor's editor writes tabs into a Python file, a shell script commits with CRLF and fails in the container, and a notebook diff is unreviewable.

Fix: Add .editorconfig, .gitattributes with text=auto eol=lf, and a formatter config.

Checked deterministically against the repository tree and a bounded set of its files: committed credentials, unpinned actions and base images, missing lockfiles and update bots, workflows that discard failures, published advisories against the declared dependencies, runtime configuration, licensing and notebook reproducibility. No language model is involved in this section.