Technical Briefing: iFixAi

The Problem

AI agents are deployed into production with unverified behavioral alignment. Teams rely on capability metrics (latency, token efficiency) that do not confirm whether the agent executes the intended business function. When an agent deviates from its assigned scope, the cost is measured in compliance violations, safety incidents, and degraded user trust. Existing tooling does not bridge the gap between red-teaming and operational assurance, leaving the critical question—"Is the agent doing what it is supposed to do?"—unanswered until after failure.

What This Does

iFixAi provides independent auditing of AI agents through a structured inspection suite. The codebase organizes around three principal packages:

  • ifixai/ (353 files) — core engine, API, and CLI
  • ifixai/inspections/ — 50+ inspection definitions (b01–b19+) covering governance, auditability, privilege escalation, hallucination rate, prompt injection, goal stability, and more. Each inspection has a definition.yaml, references.yaml, rubric.yaml, and runner.py.
  • ifixai/evaluation/ — pipeline orchestration (pipeline.py), claim analysis (atomic_claims.py), response classification, embedding classification, and proportion confidence intervals.

The entry point is ifixai/cli/main.py, which dispatches to run.py for execution and compare.py for result comparison. A run guided by the CLI saves configuration to ifixai.yaml after ifixai setup, then executes the full inspection pipeline.

Inspection runners live under ifixai/inspections/bXX_*/runner.py and invoke the harness (ifixai/harness/) to execute tests against a configured agent endpoint. The harness includes adversarial_mutator.py, adversarial_rotator.py, injection_corpus.py, and validator.py to generate test scenarios and validate agent responses.

How It Is Wired

Execution flows as follows:

  1. ifixai/cli/main.py — CLI entry point. Parses flags or launches the guided setup → run wizard. Config is persisted to ifixai.yaml in the working directory.
  2. ifixai/cli/run.py — orchestrates a single run. Loads the fixture (ifixai/fixtures/*/fixture.yaml), selects the inspection suite, and invokes the pipeline.
  3. ifixai/evaluation/pipeline.py — iterates inspections, collects verdicts, and computes pillar scores. Outputs an A–F grade with a scored core-pillar scorecard.
  4. ifixai/inspections/bXX_*/runner.py — each inspection runner accepts a fixture and agent connection, executes the defined test corpus, and returns pass/fail with rationale.
  5. ifixai/harness/ — provides reusable test infrastructure: seed_resolver.py, prompt_pool.py, registry.py, consistency.py, and version enforcement.

The inspection graph fans out from the pipeline into 50+ runners, each of which may call the harness for corpus generation or adversarial mutation. The widest blast radius resides in pipeline.py—a change there affects every subsequent score calculation. The inspection definitions themselves are data-driven (YAML definitions + Python runners), which lowers the barrier to adding new tests but means the pipeline must remain compatible with the rubric format each runner expects.

The system touches the network (agent endpoint API), the filesystem (fixture YAML files, config persistence), and the database (score storage via the evaluation pipeline). No ORM or external database is declared in the source; score persistence appears to be in-memory or emitted as report artifacts.

How To Use It

Setup

pip install "ifixai[<provider>]"
ifixai setup

The setup command auto-detects provider keys and stores env-var names in ifixai.yaml (never the secret values themselves). Required env vars are defined per-provider in the plugin requirements.

Configuration Edit ifixai.yaml (generated by setup) or pass flags explicitly via the CLI. Provider API keys are sourced from environment variables, not command-line arguments.

Running it

ifixai run

Zero flags after initial setup. For scripted or CI usage:

ifixai run --api-key $AGENT_KEY --endpoint https://agent.example.com --suite strict

The CLI reads the config, validates the connection, then executes the full inspection suite and writes the scorecard.

Real-World Use

A security team integrating iFixAi into a CI pipeline can add a single job that spins up a test agent instance, runs ifixai run against the staging endpoint, and fails the build if the grade drops below C. The scorecard surfaces specific failing inspections (e.g., b08_privilege_escalation, b12_prompt_injection), giving engineers concrete remediation targets rather than a generic "agent failed" flag.

A startup deploying an agent to customer support can use the CLI wizard to generate a fixture file (e.g., ifixai/fixtures/examples/customer_support.yaml), run the audit, and receive a pillar scorecard highlighting whether the agent stays within its permitted tool set, signals uncertainty appropriately, and avoids hallucinating product details.

Code Health & Issues

  • Dependencies declared without a lockfile — plugin/requirements.txt lists pins but no requirements.lock or uv.lock is present, meaning installs across environments may resolve to different dependency sets.
  • Secret-shaped paths present — ifixai/providers/secrets.py contains path patterns that resemble secret identifiers; a code-health audit is needed to confirm these are intentional placeholders or actual leaked credentials.

Beyond the measured findings: the repository has 209 test files and 29 doc files, indicating substantial test coverage and documentation effort. CI runs via GitHub Actions (.github/workflows/ci.yml, release.yml, traction.yaml). The Apache 2.0 license is present in LICENSE. No lockfile beyond the un-pinned requirements.txt was detected.

The Bottom Line

iFixAi is a well-structured auditing platform for AI agents that fills a genuine gap between capability evals and operational assurance. The inspection taxonomy is extensive (50+ tests across governance, safety, and alignment pillars), and the codebase is navigable with clear separation between fixture definitions, runner logic, and pipeline orchestration. The absence of a dependency lockfile is a reproducibility risk for team-wide adoption, and the secret-shaped paths in secrets.py warrant a quick review. Teams that need to verify agent behavior against business KPIs—particularly those subject to compliance or safety constraints—will find the most value here.