The Problem

AI agents need a reliable, structured way to invoke dbt commands, read model metadata, and diagnose job failures. Without a curated set of “Agent Skills” the prompt‑to‑dbt translation is ad‑hoc, leading to brittle automation and excessive manual scripting.

What This Does

The repository ships a library of markdown‑based skill definitions (skills/…/*.md) plus a Python test harness under evals/. The harness validates that each skill can be exercised against realistic dbt scenarios (e.g., evals/scenarios/dbt-job-failure/). Core runtime lives in evals/src/skill_eval/, with the CLI entry point evals/src/skill_eval/cli.py. Functions such as run(), grade(), and report() orchestrate scenario loading (load_scenario in models.py), execution (Runner in runner.py), and result summarisation (Reporter in reporter.py).

How It Is Wired

Execution begins at run (line 216 of cli.py). The call chain is:

  1. cli.run → creates an isolated run directory (create_run_dir) – filesystem write.
  2. Loads the selected scenario via load_scenario (models.py).
  3. Instantiates Runner (runner.py) which: Calls external commands (subprocess.run) to invoke dbt and makes a single outbound HTTP request (no timeout) – see Runner._get_claude_credentials. Performs file diffs (_compare_dirs) and changed‑file detection (_find_changed_files).
  4. Results are passed to Reporter.generate_report → Reporter.save_report writes report_file.
  5. If grading is requested, Grader.build_grading_prompt reads the scenario file, calls call_claude_grader (external network request), and parses the YAML response (parse_grade_response).

The internal call graph shows Runner invoked from 28 distinct locations, making it the highest‑blast‑radius component. SkillSet, Scenario, and parse_grade_response also have wide reach (≥ 7 callers). No circular imports are present, simplifying static analysis, but deep nesting (max indent 9) in runner.py, cli.py, and reporter.py makes the control flow hard to follow.

How To Use It

# Clone the repo
git clone https://github.com/moses-y/dbt-agent-skills
cd dbt-agent-skills

# Install Python dependencies (uses uv if available, else pip)
uv sync   # reads evals/pyproject.toml
# or:
pip install -r <(uv pip compile evals/pyproject.toml)

# Run the CLI against a scenario (example: dbt-job-failure)
python -m evals.src.skill_eval.cli run --scenario dbt-job-failure

Configuration – the only required external config is the optional .env for the dbt-job-failure scenario (evals/scenarios/dbt-job-failure/.env.example). No secret files are committed.

Grading – to invoke the Claude‑based grader, ensure the environment variable CLAUDE_API_KEY is set; the key is read in Runner._get_claude_credentials.

Real‑World Use

A CI pipeline can call python -m evals.src.skill_eval.cli run --scenario dbt-docs-arguments after a dbt PR to verify that the generated documentation matches expectations and to surface any breaking changes before merge.

Code Health & Issues

  • High – CI never runs tests – .github/workflows contain no test step despite 9 test files. Add a pytest step.
  • Medium – No dependency‑vulnerability scan – workflow lacks dependency-review-action or similar. Add it to catch known CVEs.
  • Medium – Outbound request lacks timeout – runner.py makes a network call without a timeout, risking hangs. Pass timeout= or use a configured requests.Session.
  • Low – Workflow jobs missing timeout-minutes – codeowners-check.yml runs without a bound, allowing six‑hour defaults. Set a realistic limit.

Additional observations: the repo declares dependencies but provides no lockfile (evals/pyproject.toml), which impacts reproducible builds. The Python modules have zero import edges, indicating a flat architecture but also a lack of shared utilities, contributing to duplicated logic and deep nesting.

The Bottom Line

The repo offers a well‑organized collection of dbt Agent Skills and a functional Python harness for scenario testing. Strengths are the clear skill markdown layout and the end‑to‑end CLI flow. Weaknesses are the high cognitive load in core modules, missing CI test execution, and lack of safety guards (lockfile, request timeouts). It is suitable for teams that need a ready‑made skill set for dbt‑enabled agents and are prepared to tighten the CI and refactor the deeply nested runner logic.