The Problem

Enterprises that expose AI coding assistants (e.g., Claude Code, Cursor) lack a unified way to observe agent actions, benchmark security posture, and automatically block unsafe behavior. Without this, policy violations and data‑exfiltration attacks go undetected until damage is done.

What This Does

ADR implements an end‑to‑end security stack for enterprise agents.

  • Observability – Sensor/adr_sensor/ collects raw telemetry, normalises it to the schema defined in Sensor/adr_sensor/schemas/agent_event_schema.py, and writes JSON logs to the configured output directory (see cli.py).
  • Benchmark & Detection – Detection/benchmark/agentdojo/ ships 300+ synthetic tasks and 133 “MCP servers” that simulate real‑world tooling. The dual‑agent detector lives in Detection/guardrail/adr_agent/adr_baseline.py and orchestrates a high‑recall triage stage followed by a deeper reasoning stage (Detection/benchmark/agentdojo/benchmarks/agentdojo/agent_pipeline/).
  • Evaluation workflow – docs/REPRODUCIBILITY.md describes how to inflate the packed benchmark (Detection/benchmark/benchmark_pack.py), run the detector, and generate the paper figures.

How It Is Wired

Execution starts at Sensor/adr_sensor/cli.py:34 (main). main parses CLI flags, creates the output directory, and calls cmd_inflate, which writes a large JSONL file (Detection/benchmark/adr_bench_20251017_151604.jsonl).

From there the control flow follows the internal call graph:

  • log_action (called 230 times) and log_operation (135 times) are the primary logging hubs; they reside in Detection/context_providers/source_codes/mcp_servers_0/api_platform/api_platform.py.
  • The detector’s entry point adr_baseline.py invokes get_triage_model → get_reasoning_model, each of which loads a language model (open‑AI or Anthropic) and then calls track_operation (26 calls) to record timing.
  • Core benchmark execution occurs in Detection/benchmark/agentdojo/benchmarks/agentdojo/agent_pipeline/agent_pipeline.py, which iterates over tasks defined in Detection/benchmark/agentdojo/benchmarks/agentdojo/default_suites/v1/*/user_tasks.py. Each task ultimately calls ground_truth (≈ 90 calls) and utility helpers that parse timestamps (Sensor/adr_sensor/utils/timestamp_utils.py).
  • All file I/O (≈ 40 functions) is confined to the sensor’s cli.py (output dir creation) and the benchmark loader (load_suites.py). No persistent database is used; the only external side‑effects are model inference calls (13 functions) and a single filesystem write during inflation.

The most‑connected modules are Sensor/adr_sensor/observer (imports 9 others) and the benchmark agent creation helpers (Detection/benchmark/agentdojo/benchmark_agents/.../creation.py). Their high instability scores (0.8‑0.9) indicate that changes here cascade widely.

How To Use It

# Clone the upstream repository (the open‑source fork)
git clone https://github.com/moses-y/ADR
cd ADR

# Install Python dependencies with UV (declared in the two pyproject.toml files)
uv sync -p python3.11   # runs in Detection/ and Sensor/ workspaces

# Set API keys for the LLM back‑ends required by the detector
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...

# Run the sensor to collect a sample trace (writes to ./out)
python Sensor/adr_sensor/cli.py collect --output-dir out

# Inflate the benchmark data and run the default detector
cd Detection
python -m detection.main_benchmark   # uses the dual‑agent detector

Configuration: model keys are read from the environment; the sensor’s output directory is a required CLI argument. No additional config files are mandatory for a smoke test.

Real‑World Use

A CI pipeline for an internal code‑assistant could invoke cli.py collect on every pull request, store the resulting JSON in an artifact store, and run main_benchmark as a gate. If log_action reports a “tool‑use violation”, the pipeline aborts the PR automatically.

Code Health & Issues

Measured static findings (92 total)

  • Medium – Broad exception handling – 20 files (e.g., Sensor/adr_sensor/observer.py).
  • High – Deep nesting – 31 files (e.g., Detection/guardrail/adr_agent/adr_baseline.py).
  • High – Duplicated code blocks – 2407 repeated 6‑line snippets across 130 files (benchmark agent creators).
  • Medium – File opened without context manager – 2 occurrences (f_secure_agent_creation.py).
  • High – Oversized files – 6 files > 750 lines (e.g., adr_baseline.py).

Repository‑wide health audit

  • Critical – Cloud credential file committed at Detection/context_providers/source_codes/mcp_servers_2/jira_server/environment/.aws/credentials.
  • High – GitHub Actions not pinned to commit SHAs (.github/workflows/*).
  • High – Hard‑coded JWT signing key in Detection/context_providers/source_codes/mcp_servers_2/jira_server/environment/config.py.
  • Medium – No dependency‑vulnerability scan in CI.
  • Medium – No pre‑commit secret‑detection hook.
  • Medium – Large benchmark JSONL (9.6 MiB) should be stored via Git LFS.
  • Low – Workflow jobs lack timeout-minutes.

Additional observation: the repository declares dependencies only in pyproject.toml files; there is no lockfile, making reproducible builds non‑deterministic.

The Bottom Line

ADR provides a concrete, production‑tested pipeline for observing, benchmarking, and detecting unsafe agent behaviour, with clear entry points and modular sensor/detector separation. However, the codebase suffers from duplicated logic, deep nesting, and several serious security hygiene lapses (exposed credentials, hard‑coded secrets). Engineers should address the high‑severity health issues before extending the platform, and refactor the most unstable modules to reduce blast radius.